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Abstract. Let (G, G) be a reductive dual pair over a local field k of characteristic 0, and denote 
by V and V the standard modules of G and G, respectively. Consider the set MaxHom(y, V^) of 
full rank elements in }lom{V,V), and the nilpotent orbit correspondence C C and 0(0) C g 
induced by elements of MaxHom(l/, V) via the moment maps. Let (tt, be a smooth irreducible 
representation of G. We show that there is a correspondence of the generalized Whittaker models 
of TV of type O and of ©(tt) of type 0(0), where 0(7r) is the full theta lift of tt. When (G, G) is 
in the stable range with G the smaller member, every nilpotent orbit O C is in the image of the 
moment map from Max Hom(V^, V). In this case, and for i non- Archimedean, the result has been 
previously obtained by Moeglin in a different approach. 



Contents 

1. Introduction and main result 2 

2. Classical groups as isometry groups of Hermitian modules 6 

2.1. Hermitian D-modules 6 

2.2. Isometry groups 7 

3. Nilpotent orbits of classical groups 

3.1. Nilpotent orbits and s[2-triples 

3.2. e-Hermitian Young tableaux [9 

3.3. Generalized Whittaker models associated to nilpotent orbits 11 

4. A realization of generalized Whittaker models: k Archimedean 12 

4.1. Norms on G 12 

4.2. Inequalities regarding norms 13 

4.3. Some spaces of rapidly decreasing functions on G 14 

4.4. Realizing a generalized Whittaker model on A^\G 17 

5. Dual pairs and theta lifting of nilpotent orbits 19 

5.1. Reductive dual pairs of type I 19 

5.2. Moment maps and lifting of nilpotent orbits 20 

5.3. Lifting of s[2-triples 22 

5.4. Isomorphism of g_i © g_i with a symplectic subspace W^^^ of Hom(y, V) 23 

6. Generalized Whittaker models and Howe correspondence 24 

6.1. The smooth oscillator-Heisenberg representation associated to W^^^y 24 

6.2. The main result and the key proposition 26 

6.3. Strategy for the key proposition 27 

6.4. The inductive step: relating covariants of ^r),(r+i) with ^r-i).(T-) 

6.5. From '3^^r).{r+i) to =^,^: proof of Proposition 38 
Appendix A. Some facts on vector valued distributions |40 



2000 Mathematics Subject Classification. 22E46 (Primary). 

Key words and phrases. Local theta lifting, generalized Whittaker models, nilpotent orbits. 
Both authors are supported by MOE2010-T2- 2-113. 

1 



A.l. Vector valued distributions 40 

A. 2. Transverse order of a ^^-distribution 41 

A. 3. Transverse jet bundle 41 

A. 4. Transverse symbols 42 

References 42 



1. Introduction and main result 

Let i be a local field of characteristic 0, for which we fix a non-trivial unitary character ip. Let 
G be a reductive group over i, g its Lie algebra, on which we fix an Ad G-invariant non-degenerate 
bilinear form k. Let 7 = {X, H, Y} C g be an s^-triple, that is 

[H, X] = 2X, [H, Y] = -2Y, [X, Y] = H. 

Set Qj = {Z S g \ad{H)Z = jZ}, for j € Z. Then, from standard s[2-theory, we have a finite 
direct sum g = ®jez0j- Define the Lie subalgebras u = ©j<-20j, n = ©j<-i5i) P = ®j<o9j and 
m = go- Let U, N, P, and M be the corresponding subgroups of G. Thus U = expu, N = expn, 
P = {peG\ Ad{p)p C p} and M = {m G G I Ad{m)H = H} . Let 

(1.1) x^{^^vZ) = ^{k{X,Z)), VZgu. 

Since k(X, [u, u]) = 0, x-y defines a character on U . Let Mx = {m S M \ Kd{m)X = X}. Then it 
is known p| Section 3.4] that 

Mx = G^:={g(^G\ kd{g)X = X, Ad{g)Y = F, kd{g)H = H} . 

In particular Mx is reductive. For the moment assume that g_i 7^ 0, or equivalently u C n. In this 
case ad(X)|g_-^ : g_i — > gi is an isomorphism, and we may define a symplectic structure on g_i 
by setting 

(1.2) K-i{S,T) = K{ad{X)S,T) = k{X,[S,T]), for aU 5, T G g_i. 

The associated Heisenberg group is N/Kerx-y, with center U/KerX'y, where Kerx7 denotes the 
kernel of x-y- Then, according to the Stone- von Neumann theorem, there exists a unique, up to 
equivalence, smooth irreducible (unitarizable) representation (Px-y^'^x-y) ^ such that U acts by 
the character x-y- Since Mx preserves 7, it is well-known ^31j that there exists a central cover of 
Mx , to be denoted by , and a representation of a semi-direct product My.^ x on =5^^^ which 
extends the representation p^^ of A^. (Depending on the circumstances, this central cover may 
be taken to be either the group Mx itself, or a double cover.) We refer to the representation of 
My.^ IX A^ on ^-^^ as the smooth oscillator-Heisenberg representation. If g_i = 0, then we define 
My.^ to be just Mx- For notational convenience, we also denote by (Px-yT^x-y) 1-dimensional 
representation of N = U given by the character x-y- 

In this article, a smooth representation of a reductive group over i will mean a smooth represen- 
tation in the usual sense for i non-Archimedean, namely it is locally constant, and a Casselman- 
Wallach representation for k Archimedean. The reader may consult [421 Chapter 11] for more 
information about Casselman-Wallach representations. 

Definition. Fix an s[2-triple 7 = {X,H,Y} C g. Let (tt,'^) be a smooth representation of G. We 
define the space of generalized Whittaker models of vr associated to 7 to be 

(1.3) Wh^{7T) = RomN{y,yx-,)- 
Note that Wh..y(7r) is naturally an M^^-module. 
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From the well-known results of Jacobson-Morozov and Kostant [2l Chapter 3], the map 7 = 
{X, H,Y} O = Ad G ■ X yields a 1-1 correspondence between 

J Conjugacy classes of 1 J Nilpotent Ad G-orbits 1 

\ s[2-triples ingj \ O C Q j ' 

If the conjugacy class of an s[2-triple 7 corresponds to a nilpotent orbit O C we say that 7 is an 
5l2-triple of type O. It is clear that given two conjugate s[2-triples 7, 7', there exists an isomorphism 
(j) : Wh^(7r) — > Wh^/(7r) that intertwines the action of and ,. For this reason, sometimes 
we will abuse notation and denote Wh^(7r) just by Whc)(7r) and we will call it the space of generalized 
Whittaker models associated to O, or the space of generalized Whittaker models of type O. Note 
that this definition is (essentially) equivalent to the one given in | i43| |23| . 

The study of Whittaker and generalized Whittaker models for representations of reductive groups 
over local fields evolved in connection with the theory of automorphic forms (via their Fourier 
coefhcients), and has found important applications in both areas. See for example |34 | 126 1 [T5 | 113 1 143 | 
|40] , and a recent article of Jiang [12j discussing its role in a general theory of periods of automorphic 
forms. From the point of view of representation theory, the space of generalized Whittaker models 
may be viewed as one kind of nilpotent invariant associated to smooth representations. Another 
nilpotent invariant is the wave front cycle: 

WF(7r) = co(^)[0], 
nilpotent 

defined by Harish-Chandra in the non-Archimedean case [6] and by Barbasch and Vogan in the 
Archimedean case |lj. For k non- Archimedean, Moeglin and Waldspurger ||23] have established that 
WF(7r) completely controls the spaces of generalized Whittaker models of interest, namely, if O is 
a nilpotent orbit which is maximal in the wave front set of vr | 10| , then 

ce'('?r) = dim Who (vr). 

For k Archimedean, the corresponding phenomenon is not yet (fully) understood, except for the 
representations with the largest Gelfand-Kirillov dimension |37| |20| and unitary highest weight 
modules [44j. 

On the other hand, when k is Archimedean, there is another nilpotent invariant defined by Vogan 
[38], which is called the associated cycle: 

AC(^)= do{7r)P]. 

O cs 
nilpotent 

Here gc = © s is the complexified Cartan decomposition. By the work of Schmid and Vilonen 
[32], we know that the wave front cycle and the associate cycle determine each other through the 
Kostant-Sekiguchi correspondence |33|. 

Now we come to the other part of the title regarding local theta lifting. Let (G, G) be a reductive 
dual pair in the sense of Howe [7]. Let (w, '3/') be the smooth oscillator representation associated to 
the dual pair (G, G) and to the character ip oik. It is well-known that (w, ^) yields a representation 
of G X G except when G is the isometry group of an odd dimensional quadratic space, in which case 
it is a representation of a double cover of G. In this exceptional case, we shall simply redefine G 
to be the induced double cover, so as to simplify notation. Also in this case we shall only consider 
genuine representations of G, namely those representations such that the non-trivial element of the 
covering map acts by —1. For all other cases, representations are called genuine by convention. 
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Let (tt, be a smooth irreducible genuine representation of G, and consider the maximal vr- 
isotypic quotient of called the Howe quotient of vr. If it is non-zero, it is of the form Y Q{'f) 
(algebraic tensor product for k non-Archimedean, and completed projective tensor product for i 
Archimedean), where Q{'f) carries a smooth representation G(7r) of G. The representation 0(7r) is 
sometimes referred to as the full theta lift (or colloquially the big theta lift) of vr. Results of Howe 
[TT] and Waldspurger [39] say that 0(7r) is an admissible representation of finite length, moreover, 
with the possible exception when i is a dyadic field, Q{tt) has a unique irreducible quotient ^(vr) 
called the (local) theta lift of vr. In this article we shall focus on the full theta lift G(7r), and shall 
thus place no restriction on the residue characteristic of i. 

Let V, V, be the standard modules of G, G, respectively; and consider the moment maps ip, (p, 
from Hom(y, V) to g and g, respectively: 



Hom(y, V) 








If T G Hom(y, V), then <p{T) is nilpotent if and only if (p{T) is nilpotent. As usual, this yields a 
notion of correspondence for nilpotent orbits O C g and O C g. (In general this may not be one to 
one.) Denote by MaxHom(y, y) the set of full rank elements in Hom(y, F). Without any loss of 
generality, we assume that dimV < diml/, and elements of MaxHom(y, y) are then represented 
by injective maps from V to V . In this article, we shall only be concerned with nilpotent orbits 
O C g in the image of MaxHom(y, V) under the moment map ip, namely it satisfies 

(L4) (/?~^(0) nMaxHom(y,y) 0. 

This will be our standing assumption. We write the resulting nilpotent orbit correspondence as 
O ^ O = B{0). See Section [5j for details. 

As noted in the beginning, the space of generalized Whittaker models depends on an s[2-triple. 
To relate the two s[2-triples in g and g, we make the following 

Definition. Let 7 = {X,H,Y} C g and 7 = {X,H,Y} C g be two s[2-triples of type O and O, 
respectively. We say that T € Hom(F, V) lifts 7 to 7 if ip{T) = X, (p{T) = X and T{Vj) C Vj+i 
for all j. Here Vj = {v \ Hv = jv}, and likewise for V^+i. 

Set 

O^^^ = {T e Hom(F, V) I T lifts 7 to 7}, 

and 

(L5) Oi^?^ = O^^^y n MaxHom(y, V). 

(By Lemma 15.71 this is a single Mx x Afj^-orbit.) 

Our main result, in a slightly less concrete form than what will be proved in Section [6l is 

Theorem 1.1. Let {G,G) be a reductive dual pair, and let (u,'3^) be the smooth oscillator repre- 
sentation associated to the dual pair (G, G) and to the character tp of k. Let (tt, Y) be a smooth irre- 
ducible genuine representation ofG. Let O C Q be a nilpotent G-orbit in the image o/MaxHom(y, V) 
under the moment map ip and let O = 0(0) C Q be the corresponding nilpotent G-orbit. Then 

Whe(o)(e(7r)) ^Who(7r), 

where tt is the contragredient representation o/tt. More precisely, let 7 and 7 be two 5\2-triples of type 
O and Q{0), respectively. Then, given any T = T^^^y G O^^^, there is a surjective homomorphism 
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(pT '■ ~^ ^x-i (depending on T), inducing an action of M^- on Wh^(7r), such that 

Wh^(e(7r)) ^ Wh^(7r) 

as M^. -modules. 

Remarks: (a) Nilpotent orbits in g may be parameterized by equivalence classes of admissible e- 
Hermitian Yomig tableaux (Section l3.2p . Using this parametrization, the correspondence O 0(0) 
can be described explicitly and in simple terms. Note that the assumption in (jl.4p allows us 
to "embed" the sesquilinear Young tableau parameterizing O into the sesquilinear Young tableau 
parameterizing Q{0). See Section [5.21 

(b) One particularly important circumstance is when (G, G) is in the stable range with G the smaller 
member cf. [18]). Then every nilpotent orbit O C is in the image of MaxHom(y, T/) under 
the moment map if, and for any smooth irreducible genuine representation (tt, i^) of G, we have 
G(7r) 7^ |221I28|. In this case and for i non- Archimedean, Theorem 1 1.1 1 is due to Moeglin |21| . Our 
approach, explained in the following paragraph, is in some sense more conceptual. For the type of 
questions considered in this article, it is also well-known that substantially more effort is required 
to treat the Archimedean case. 

(c) Assume that we are in the stable range and i is Archimedean. There is a similar notion of theta 
lifting of nilpotent ETc-orbits and one similarly expects a correspondence of associate cycles. We 
refer the reader to |44| [25] for some results in this direction, and the definitive result in the recent 
paper of Loke and Ma |19| . We also refer to earlier works of Przebinda |29( l30] on correspondence 
of wave front sets. 

(d) Generically ©(vr) should be irreducible. One then obtains the space of generalized Whittaker 
models for 9{7r). For example it is expected that under the assumption of stable range, ©(vr) is 
irreducible whenever vr is unitarizable. For i Archimedean, this is established in |19) . 

(e) As a direct consequence of Theorem ll.il we have ©(vr) ^ 0, if Whc)(7r) ^ 0, for some O satisfying 
the condition in (jl.4p . 

The main ingredient in the proof of Theorem 11.11 is our description of the space of covariants 
'3^jj ^_ of the smooth oscillator representation where (JJ, X7) for the s[2-triple 7 in g is as (U, x-y) 
for the s[2-triple 7, and 7 is a lift of 7. (This is reminiscent of the computation of the Jacquet 
modules of the smooth oscillator representation by Kudla [17], though not in method.) We show it 
is isomorphic to a certain space '^{N\G; .^,7) of rapidly decreasing functions on A'^\G with values 
in where M'^^x^ is the space of the smooth oscillator-Heisenberg representation associated to 

a certain symplectic subspace W^,^ of Hom(l/, 1/). This is the key Proposition 16.41 An important 
point is that we may construct an isomorphism of 0-i®0-i with W^^^y, which depends on T € O^l"^. 
The key proposition basically says that we can define a natural surjective (matrix coefficient) map 
(using "homogeneous components" of T), from ^ to ^{N\G] M'-^ ^\ which induces an isomorphism 

Here are some additional words on the organization and contents of this article. In Section [21 we 
describe classical groups as the isometry groups of e-Hermitian D-modules, where Z) is a division 
algebra over k. In Section [3l we review the well-known parametrization of nilpotent orbits in the 
classical Lie algebras, following the book by Collingwood and McGovern [2]. We also introduce 
the generalized Whittaker models associated to the nilpotent orbits. In Section [H we introduce 
some Frechet spaces of functions on G to realize the space of generalized Whittaker models for i 
Archimedean. In Section [5l we recall the notion of lifting of nilpotent orbits in the setting of dual 
pairs (via the moment maps), and we describe the fine structure of lifting for those orbits O and 
O which correspond via injective maps from V to V . We also prove an explicit isomorphism of 
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0-1 © 0-1 with a symplectic subspace W-^^-y of }iom.{V,V). In Section [HI we relate the generahzed 
Whittaker models of tt and 0(7r). As mentioned previously, its main ingredient is the description of 

the space of covariants ^ of the smooth oscillator representation To arrive at this, we make 

^ 1X7 

extensive use of the gradation in the standard modules V and V given by the neutral elements H, 
H of the two s[2-triples 7 C g and 7 C g of type O and 0(O), respectively. On the one hand, 
it gives rise to totally isotropic subspaces and thus convenient realizations of '3^. On the other 
hand, it facilitates an inductive argument based on the heights of the gradations. Together with the 
isomorphism of g_i0g_i with W^^iy, this implies the relationship between the generalized Whittaker 
models of vr and 0(7r). 

Acknowledgements: the authors thank W. T. Gan, D. Jiang, and B. Sun for their interests and 
comments. 

2. Classical groups as isometry groups of Hermitian modules 

2.1. Hermitian D-modules. Let i be a local field, | • | its absolute value, and let be a fix 
non-trivial unitary character of k. Let D be one of the following division algebras over k: the field 
i itself, a quadratic field extension of i or the quaternion division i-algebra. Observe that D comes 
equipped with a canonical involutive anti-automorphism (the identity map, the non-trivial Galois 
element, or the main involution, respectively) which we will denote by x 1— )■ x. Throughout this 
article, we will only consider finitely generated modules over D. 

Let V and W be two right D-modules. We will denote the set of right D-module morphisms 
from y to by 

Hom^(F, W) = {T:V ^W\ T{via + V2h) = T{vi)a + T{v2)h for all vi,V2eV,a,he D}. 

li V = W, we will denote this set by FindDiV). Set 

GL{V,D) = {T G EndD(y) | T is invertible}. 

When it is clear from the context what the division algebra is, we may just omit D in various of 
these notations. 

Let V' be the set of right D-linear functionals on V. There is a natural left D-module structure 
on V' given by setting 

{aX){v) = aX{v), for all a e D, v e V, and A € V' . 

Observe that with this structure, W ^^y' is naturally isomorphic to }iom£)(V,W) as a i- vector 
space. Given T E Hom£)(y, W), we will specify an element in Hom£)(VF', V') (analogously defined), 
which we will also denote T, by setting {\T){v) := X{Tv), for v &V and A G W. This correspon- 
dence gives rise to natural isomorphisms between End£)(y) and End£)(y), and between GLoiV) 
and GLi5(^')- 

Definition 2.1. Let e = ±1. We say that {V,B) is a right e-Hermitian D-module if F is a right 
D-module and B is an e-Hermitian form, i.e., B : V x V — > D is a map such that 

(1) B is sesquilinear: for all vi, V2, V , a, b G D, 

B{vi,V2a + V36) = B{vi,V2)a + B{vi,v^)h and B{via + ^2^, "^3) = aB{vi,v:i) + bB{v2,V5). 

(2) i? is e-Hermitian: 

B{v, w) = eB{w, v) for all v,w . 

(3) B is non- degenerate. 
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Given a right e-Hermitian D-module {V,B), we may construct a left e-Hermitian Z)-module 
(y*, B*) in the fohowing way: as a set, V* will be the set of symbols {v* \ v G V}. Then we give to 
V* a left I?-module structure by setting, for all f, it; G F, a G I?, 

V* +w* = (y + w)* and av* = {va)*. 

Finally, we set 

B*{v*,w*) = B{w,v) for all -u, It; G y. 

In an analogous way, we may define the * operation on left e-Hermitian D-modules. Then V** is 
naturally isomorphic with V. Given T G EndoiV), we define T* G Endz)(l^*) by setting v*T* := 
(Tv)*. With this definition, it is easily seen that {TS)* = S*T*, for all 5, T G EndD{V). Therefore 
the map g i— > (g*)^^ defines a group isomorphism between GLdIV) and GL£)(V*). 

Observe that the form B induces a left -D-module isomorphism B^ : V* — > V' given by 
B\v*){w) = B{v,w) for V, w E V. In what follows, we will make implicit use of this map to 
identify these two spaces. With this identification, for any T G End/)(^), we can think of T* as 
an element in En.dD{V) defined by v*{T*w) := {v*T*)(w), i.e., T* is defined by the usual condition 
that 

B{v, T*w) = B{Tv, w) for all v, we V. 

A D-submodule E C V is said to be totally isotropic if B\exE = 0. If is a totally isotropic 
submodulc, then there exists a totally isotropic submodule F C V such that B\(^e®f)x{e®f) is 
non-degenerate. If wc set 

U = {E® F)^ ■.= {ueV\ B{u, w)=0 for all w e E ® F}, 

then V = E (B F (B U , and B\ijxu is non-degenerate. In this case we say that E and F are totally 
isotropic, complementary submodules. Observe that then B^\f* '■ F* — y E' is an isomorphism. As 
before we will make implicit use of this isomorphism to identify F* with E' . 

2.2. Isometry groups. Given a right e-Hermitian D-module {V,B), we define its isometry group 

G{V,B) = {ge GL(y) | B{gv,gw) = B{v,w) for all v, w e V}. 

When there is no risk of confusion regarding B, we will denote this group just by G{V) or even just 
as G. Observe that if g e G, then g* = g~^. 

Associated to the group GL(y) we have the Lie algebra gl{V) = End{V) with bracket [T,S] := 
TS — ST, for all T, S E qI{V). Similarly, associated to the group G we have the Lie algebra 

= {T G 9l{V) I B{Tv, w) + B{v, Tw) = for all v,weV] 

= {TGgi(y)|r* = -r}. 

We define a bilinear form on q by 

k{T,S) =Tr(T*S)/2, 

for all T, S G 0. Here Tr(r*5) is the trace of T*S as a i-linear transformation. This bilinear form 
is non-degenerate and invariant with respect to the adjoint action of G on g. 

Remark 2.2. In the literature, the isometry groups G{V, B) are called type I. Given a division algebra 
D, one may consider the algebra = D (B D, which has the canonical involution (a, b) = (6, a), for 
a, b & D. Then one may similarly define right e-Hermitian B-modules (e = ±1) and their isometry 
groups which are easily seen to be isomorphic to GL{V,D), where V are right D-modules. They 
are called type II. With this modification (from D to ED), all results in this article, which are stated 
for type I dual pairs, carry over to the case of type II dual pairs. 
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3. Nilpotent orbits of classical groups 



3.1. Nilpotent orbits and s[2-triples. In order to set up notation, and facilitate the exposition 
of the reminder of this article, we will review the basic structural results on nilpotent orbits and 
s[2-triples in q. The exposition given here is based on the book of Collingwood and McGovern |2|. 

Let X G g be a nilpotent element. Then, by the Jacobson-Morozov theorem, there exists an 
s[2-triple 7 = {H,X,Y} C 0, containing X as the nilpositive element, namely 

[H, X] = 2X, [H, Y] = -2Y, [X, Y] = H. 

Let Qi = {Z £ Q I ad{H){Z) = iZ}, for i G Z. Then, from standard s[2-theory, we have that 

(3.1) = 00i. 
Now let 

P = 00i, P = 00i, 

j<0 j>0 

and set 

P={gGG\ Ad(^7)p Cp}, P={geG\ Ad(g)p C p}. 

P (resp. p) is called the Jacobson-Morozov parabolic subgroup (resp. parabolic subalgebra) as- 
sociated to X. Observe that P (resp. p) is a parabolic subgroup opposite to P (resp. parabolic 
subalgebra opposite to p). (The subgroup P depends only on X, but the subgroup P depends on 
the choice of s^-triple 7 containing X.) Let M = {m € G\ Ad{m)H = H}. Its Lie algebra is 
m = {Z G I ad{Z)H = 0}, which is exactly 0o. Let 

= 0« " = 

i<-l i<-2 

Observe that if Z € n, then Z is nilpotent and hence exp Z = Xlj^o -^''/i' ^ ^^^^ defined element 
in End(y). Let N = expn = {expZ | Z G n} and U = expu = {expZ | Z G u}; then U, N are 
subgroups of G, and P = MN. Similarly we have the Levi decomposition P = MN, with N the 
unipotent radical of P. 

For i G 7j, let Vi = {v (z V \ Hv = iv}. Then, again from standard s[2-theory, 

(3.2) ^ = 0^- 

We may also characterize M as the set of m G G that preserves the direct sum decomposition given 
in (|3.2p . Now observe that, since H* = —H, B\v(^xVo is non-degenerate and B establishes a perfect 
pairing between Vt and V-i for all i > 0. Using this pairing we define for any T G End(T^) a map 
T* G End(y_j) given by 

B{Tv, w) = B{v, T*w), for aU veVi,we V^i. 

It follows immediately that, if 5 G GL(^), then B{gv, {g*)^^w) = B{v,w), for all u G V^, i« G V-i. 
From this we conclude that there is an embedding of GL(l/i) into M for all i > 0. We will denote 
the image of this embedding by Mj. Proceeding in a similar manner we may also define a natural 
embedding of G{Vo, B\voxVo) ii^to M, whose image we will denote by Mq. Then 

(3.3) M = llMi^G{Vo)xYlGL{Vi). 

i>0 i>0 
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Set = Span^{X, H,Y} C Q. We have the g^-isotypic decomposition 

I 

(3.4) F = 0yT'*^, 

i=i 

where V^'*^ is a direct sum of irreducible tj-dimensional g^-modules, and ti > t2 > ■ ■ ■ > ti > 0. 
Let V^'''^ = V^'^i n Vi. It is nonzero if and only if |i| < tj and i = tj — 1 (mod 2). Then it is clear 
that 

(3.5) 

i,tj>l»l 

Using standard results from the representation theory of sl2, we have that for all i > 0, the map 
(X|v"_.)* : V-i — > Vi is invertible. (Here we are using the convention that (Xlvg)" is the identity 
map on Vq.) The statement remains true for i < if we interpret a negative power of an invertible 
map as the positive power of its inverse. 

Now define a Hermitian form Bi on Vi by setting Bi(v,w) = B{X\y^v,w) for all v, w Vi. 
Since B establishes a perfect pairing between Vi and V-i, it is clear that Bi is non-degenerate. The 
analysis applied to {V,B) applies equally to {V^'^^ , B'^'^^), where B'^'^^ is the restriction of B to 
, Thus Bi is in fact non-degenerate when restricted to any V^' ^ . We will denote by B^' ^ the 

7 t ■ 

restriction oi Bi to V^ ' \ The following result is also clear: 
(3.6) 

U-tj + l<i<i + 2<tj-l, then Bi+2{Xv, Xw) = -Bi{v, w) for all v, w € F^'*' • 

In particular, all the BJ'^^'s are determined by BJ'^^^ (on V^.'j'^, the space of highest weight vectors 
in l^T'*i). 

Let Mx = {m € M \ Ad{m)X = X}. The following result is due to Kostant. 
Theorem. {[2^ Section 3.4]) Let Gx = {9 Ad{g)X = X}. Then 

Gx = MxNx, 
where Nx = {n £ N \ Ad{n)X = X}. Furthermore 

Mx = G^:={gGG\ Ad{g)X = X, Ad{g)H = H, Ad{g)Y = Y}. 

We now have all the ingredients to give a description of Mx in the spirit of the description of M 
given in equation (13. 3h . Observe that Mx acts on Vjjl.'i preserving Bj',j_-^. From this observation 

and equation (j3.6p we conclude that there is an embedding of Giy?'\,By\) into Mx- Let Mx t -i 

3 3 ' 

be the image of this embedding. Then 

I I 

(3.7) Mx = n Mx,t,-i = n GiV,^;-i)- 

j=i i=i 

3.2. e-Hermitian Young tableaux. A partition of n is a tuple [di, . . . ,dk] of positive integers 
such that (ii > ^2 > . . . > (i/fc and di + ■ ■ ■ + = n. Partitions of n are frequently represented 
by Young diagrams in the following way: given a partition d = [di, . . . ,dk] we construct a left- 
justified array of empty boxes such that the i-th row has di boxes. This array is the Young diagram 
associated to d. Another way of describing a partition d = [di, . . . , dk] of n is using the exponential 
notation d = [f^,... ,t^/], which means that the number tk appears i^ times (the multiplicity) in 
the partition, and ti > ■ ■ ■ > ti > 0. 
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Definition 3.1. A sesquilinear Young tableau is a pair r = {d^,{V^,B^)) such that = 
[t^-^, . . . , t\'] is a partition of n, and (V^ ,B^) is an assignment, for each 1 < j < of an ej-Hermitian 
module {V^.'j'i, B^_^'^_^-^) of dimension ij. 

Example 3.2. The following picture represents a sesquilinear Young tableau F = (d^, {V^ , B^)), 
where d^ = [3^ 2^ 1^] is a partition of 14, and V'g ' , Vj^ ' , ' have dimensions 2, 3 and 2, 
respectively. 

Let (pm,^^) be the irreducible representation of s[2 of dimension m, and fix, for each m, a non- 
degenerate invariant bilinear form (•, •)m on i™. Recall that this form is unique up to scalar and 
that (/i,/2)m = (— 1)™'^^(/27 /i)m) for all /i, /2 € i"*. This follows from the analysis surrounding 
equation (|3.6p . 

Given a sesquilinear Young tableau F = (d^, {V^ , B^)) we will set 

(3.8) V^''^ ,= V^^%^^S^^. 
On this space we define a (— l)*J~^ej-Hermitian form B^'^^ by 

(3.9) S^'*^ (^;i /i, t;2 ® /2) = B^^;%{vi,V2){hj2)t, , ^^i, ^2 G 1^^^:^^, and h, h G i*^' • 

Definition 3.3. Let F = {d^,{V^,B^)) and $ = (d*,(F*,S*)) be two sesquilinear Young 
Tableaux. 

(i) We say that F and $ are equivalent if d'" = d* = [t^^ , . . . , tj'] and (V^ , B^ ) is isomorphic 
to(y*:*;,i3,'^l*^Jforanj = l,...,/. 

(ii) We say that F is e-Hermitian if , i?'"'*^ ) is an e-Hermitian module for all j = 1, . . . ,1. 

(iii) Given an e-Hermitian module {V,B), we say that F is admissible for {V,B), or {V,B)- 
admissible, if (©jF'"'*-' , ©^5^'*^ ) is isomorphic to {V,B). 

Recall that given an s[2-triple 7 = {X, H, Y} C there exists ti > . . . > ti such that V = (BjV^'^J . 
Using this decomposition we may define an e-Hermitian Young tableaux F^ = (d'^T, {V^"/ , B^^)) by 
setting d^~' = [f^ , . . . ,t]'], where ij = dimV^,'^\, and 

(3.10) (<:^^<!:l') := (v;;%B^fi^, 

for all J = 1, ... , I. 

Methods of [2, Section 9.3] imply that this assignment gives a bijection between the set of s[2- 
triples in q up to the Adjoint action of G and equivalence classes of admissible e-Hermitian Young 
tableaux. We thus have 

Proposition 3.4. There is a 1-1 correspondence between the following sets: 

. r • • T f Equivalence classes of admissible 1 

\ Nilpotent orbits m q \ i — > < ■, ■ , n f ■ 

^ I e-nermitian Young tableaux J 

Remark 3.5. If i is the field of real numbers then Hermitian Young tableaux can be more concretely 
described by signed Young diagrams jjj Section 9.3]. 
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3.3. Generalized Whittaker models associated to nilpotent orbits. Let 7 = {X, H, Y} C g 

be an s[2-triple. As in the introduction, we define the character x-y of U, by X7(exp Z) = iP[k{X, Z)) 
for all Z G u; and a symplectic structure on g_i by setting 

K-i{S,T) = K{ad{X)S,T) = k{X, [S,T]) for aU S,T e Q-i. 

(Note the similarity between this definition and the definition of the forms Bi on Vi in Section 
13.11 ) Let be the Heisenberg group associated to the symplectic space (g-i,K_i). That is 
= 0_i X i, {0} X i is central, and (r,0)(5,0) = (T + S,k{T,S)/2) for ah T, S € 0_i. Then, 
according to the Stone-von Neumann theorem, there exists a unique, up to equivalence, smooth 
irreducible (unitarizable) representation {p-y,J/'-y) of such that the center of acts by the 
character ip. Here smooth means that it is locally constant if i is non-Archimedian, and if i is 
Archimedian, (p^,^^) is the smoothing of the usual irreducible unitary representation of with 
the central character ip. 

Let a-f : U — > k be the map given by a-f{exp Z) = k{X,Z), for all Z € u. It is standard 
to check that defines a surjective group liomomorphism. Furthermore, it extends to a group 
homomorphism : N given by 

(3.11) a^(exprexpZ) = (r,K(X,Z)), for all T G 0„i, Z G u. 

By composition, this yields a representation {Px-,i'^x-y) °^ where 5^^^ := and 

(3.12) p^^(n)v = p.y{a.y{n))v, for all n G A^, w G o5^x7- 
Observe that then, for all Z G u, w G ^xi^ 

p^^{expZ)v = Pj{0,k{X,Z))v = V'(k(X, Z)) = X7(expZ). 

In particular, if 0_i = 0, then N = U acts on the 1-dimensional space J/'^.^ by the character 
X-y- Since Mx preserves 7, it is well-known [SlJ that there exists a central cover of Mx, to be 
denoted by M^^, and a representation of a semi-direct product My,^ t< N on 5^-^^ which extends 
the representation p^^ of A^. (Depending on the circumstances, this central cover may be taken 
to be either the group Mx itself, or a double cover.) We refer to the representation (Pxt'^^Xt) 
of X A^ as the smooth oscillator-Heisenberg representation associated to x^. We remark that 
there is a notion of "smooth Frechet representations of moderate growth" for groups of the type 
M^^^ X A^. See 01 Definition 1.4.1] or fSS] Section 2]. 

Definition 3.6. Let (vr, Y^) be a smooth representation of G, and let 7 = {A, i7, 1"} C be an 
s[2-triple. We define the space 0/ generalized Whittaker models of n associated to 7 to be 

(3.13) Wh^(7r) = Hom^(r, y^^). 
Note that Wh^(7r) is naturally an M^^-module. 

We also make the following definition. 

Definition 3.7. Let O C be a nilpotent orbit. We say that 7 = {A, H, Y} is an 5l2-triple of type 
O if X £ O, where X is the nilpositive element of the s[2-triple. 

From the well-known results of Jacobson-Morozov and Kostant [2l Chapter 3], the map 7 = 
{A, H, Y} >—?■ O = AdG • X yields a 1-1 correspondence between 

J Conjugacy classes of 1 J Nilpotent AdG-orbits 1 

I s[2-triples in0j | O C Q J' 

As noted in the introduction, it is clear that given two conjugate s[2-triples 7, 7', there will be an 
obvious isomorphism (p : Wh^(7r) — > Wh-y/(7r) that intertwines the action of M^^ and ,. By 
abuse of notation, we will denote Wh^(7r) just by Whc)(7r) and we will call it the space of generalized 
Whittaker models associated to O, or the space of generalized Whittaker models of type O. 
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4. A REALIZATION OF GENERALIZED WhITTAKER MODELS: k ARCHIMEDEAN 



Let (tt, ^) be a smooth representation of G, and let 7 = {X,H,Y} C g be an s[2-triple. In this 
section we will give convenient realizations of the spaces and Wh^(7r) associated to 7 and tt. 
As we will see later in this section, the analysis required for i Archimedian is substantially more 
involved than the case where i is non- Archimedian. For this reason, we will devote this section to 
the Archimedean case and will be contented to just indicate how the analog results work in the 
non-Archimedian case. 

4.1. Norms on G. Assume that i = M or C. Let {V, By) be an e-Hermitian module of dimension 
n. Then we have the following possibilities for G = GiV): 
(1) i = M and D = R. In this case, either 



or G = Sp{n, M) if e = — 1 and n is even. 



(2) i 

(3) i 



regardless of e (= ±1) 



or G^O*(2n), if e= -1. 



G^O{p,q),p + q = n, if e = 1, 

. and D = C In this case 

G^Uip,q),p + q = n, 

. and D = H. In this case either 

G^Sp{p,q),p + q = n, if e = 1, 

(4) i = C and D = C. Then 

G ^ 0{n, C) if e = 1 or G ^ Sp{n, C) if e = -1 and n is even. 

Let V = E (B U (B F, with E and F totally isotropic, complementary submodules of maximal 
dimension. Thus U is anisotropic. Let Pe = Stab^;, the stabilizer of E. Then Pe = MeNe, is a 
Langlands decomposition of Pe, where Me = {m £ Pe \ m ■ F C F} and the Lie algebra of Ne is 

UE = }iom{U,E)®i, 

with ^ ^ {Z : F E\Z* = -Z}. It follows from this description that Me = GL{E) x G{U) 
and that G{U) is compact. Thus there exists a real inner product on U such that G{U,B) C 
G{U,B+). 

Let {ei, . . . , ei}, and {/i, ...,//} be basis of E and F. respectively, such that 

Bv{ei, fj) = Si^i-j+i, for all l<i,j <l. 

Then we can extend B^ to an inner product By on V by setting {ei, . . . , e;, /i, . . . , /;} to be an 
orthonormal basis oi E (B F, and {E ® F)-^ = U. 

Uwe set K = G{V, B)nG{V, By), then if is a maximal compact subgroup of G(y, By). Moreover, 
if we set 



a(Ai,...,A; 



then 

A = {a(Ai, Xi) I Ai G M*, for i = 1, . . . , /} 
is a maximal split torus in G{V,Bv) and, according to the Cartan decomposition, for any g G 
G{V, By), there exists ki, k2 € K, and a € A such that g = kiak2- 
Given g G G(y,Bv), let 

II5II = sup [B+{gv,gv)]^ 

B+{v,v)=l 
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be the operator norm restricted to G{V,B). Observe that if = kiak2, with ki, k2 £ K and 
o = a(Ai, A;), then 

llffll =max{|Ai|,...,|Ai|,|Air\...,|Ar'}- 
From this observation it is immediate that = and || exptX|| = || expX||* for all t > 0, and 

all X € = Lie(^). Since || • || is the operator norm in End(y) we also have that ||5i5'2|| ^ ||5i|| 115211 
for all 51, 52 S G, and hence || • || satisfies all the properties of a norm on G |4H 2. A. 2]. 

4.2. Inequalities regarding norms. Let 7 = {X,H,Y} C g be an s[2-triple, and let P = MN 
be as in Section [3.11 Assume that V = ©^^_^Vfc, and set Endj(y) = ©^^_^ Hom(Vfc, Vfc+j), where 
Vfc = if \k\ > r. Then 

2r 

End(l^) = End,{V), 

i=-2r 

and 0j = n Endj(y). Recall that, if g_i 7^ 0, then there is a symplectic structure on 0_i given 
by K-i(S,T) = Ti{X[S,T]). Let g^i = e © f be a complete polarization of 0_i with respect 
to this symplectic structure, and let Her_i(y) = {T G End_i(y) \ T* = T} . (Here Her_i stands 
for Hermitian of degree —1.) Then we have a decomposition End_i(y) = 0_i © Her_i(y) = 
e ffi f © Her_i(y). Define a norm || • on End_i(y) by setting 

\\Tc + Tf + THcr_i(y)||-l = ll^'ell + H^fH + ||7Her_i{V) II; 

where, as before, || • || is the operator norm, S e, Tf G f, and 7Her_i(v) ^ Her_i(y) are arbitrary. 
Now given T G End(y), let 

II^IIt = IjT'-ill-i + ^ ll^jll) 

where T = (BTi, with Tj G Endj(y). Then || ■ ||^ defines a norm on End(y) and, since all norms on 
a finite dimensional vector space are equivalent, there exists constants Ci, C2 > such that for all 
T G End(y) 

(4.1) Ci||T|| < ||r||^ < C2IITII. 

Recall that n = u © 0_i = u © e © f . Let ric = u © c and rif = u © f , which are ideals of n. Let 

(4.2) A^e = expnc and A^f = expnf 

be the corresponding normal subgroups of A^. Observe that for every n £ N there exists unique 
u (z U , Zj £ f and G e such that n = n(exp Zj)(exp Z^). It follows immediately that N^\N = e. 

Lemma 4.1. For all Z £ e, h £ N^, we have 

(4.3) ||n expZjl > C7i(l + |[Z||). 

Proof. Observe that h = expZi expZ2, for some Zi £ u, Z2 £ f. Hence, as an element of End(y), 
h exp Z = exp Zi exp Z2 exp Z = l + Z + Z2 + Z, for some Z £ ©^^2 End_fc(y). Therefore, 

||nexpZ||-y = ||1 + Z + Zi + Z||^ 

= 1 + ||Z|| + llZill + ||Z||^ 
> 1 + ||Z||. 

The lemma then follows from (|4.ip . □ 
Lemma 4.2. There exists constants d, C > such that for all Z £ t, n £ Nf 

||n(expZ)r'>C||n||||(expZ)||. 
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Proof. Since || • || is the operator norm on End(y), we have that 

IIZIP ItZiP'' 

(4.4) II expZll < 1 + ||Z|| + ^ + . + ^Ji- < C3(l + ||Z||)2^ 

for some constant C3 > 0. Here we have used that V = ®^^_j,Vfc. Combining equations (|4.3p and 
(fi^ we get that 

(4.5) ||n(expZ)||2- >Cf (1 + ||Z||)2- > ^||expZ||. 

On the other hand, since the restriction of || • || to G defines a norm on G, we have that ||fi|| < 
||(exp Z)~-^|| ||n(exp Z)|| = || expZ||||n(expZ)||. From this and equation (j4.5p 

II / ^\ii ll^ll C'^^ 1 1 ^ 1 1 
||n(expZ)|| > " " > ^ " " 



expZ|| C3 ||n(expZ)P^' 

and hence ||n(exp Z) ||'^'''*'^ > -j^||^i||. But now this inequahty and equation (j4.5p imply that, if we 
set C = (^)^ and d = 4r + 1, then 

||n(exp Z)\f > C'||n||||expZ||. 

□ 

Corollary 4.3. There exists constants d^, Cq > such that, for all k & K , m (z M , Z ^ t, and 

he Nf 

||n(exp Z)mk\\'^° > Co||ri|| || exp Z|| ||m||. 

Proof. By definition of || • ||, ||n(exp Z)mA;|| = ||fi(exp Z)m||. Now, from the proof of [41, Theorem 
7.2.1], we have that 

(4.6) |[n(expZ)m|| > ||m|| and ||n(expZ)m|p > ||n(expZ)||. 

Let C and d be as in Lemma 14.21 Then, if we set do = 2d+ 1 and Cq = C, we have that 
||n(exp Z)m/c||°''' > ||n(exp Z)||'^||m|| > Co||ra|| || exp Z|| ||m||. 

□ 

4.3. Some spaces of rapidly decreasing functions on G. We retain ah of the notation from 
the previous section. 

Let (Px-y^'^x-y) smooth Heisenberg representation of N associated to the character x-y of 

U, as in (13. 12^ . We first give a reahzation of ^x-y^ ^ fohows: fix g_i = e©f (a complete polarization 
of 0-i). Set ^(e) to be the space of constant-coefficient differential operators on e, and let 

^(e) = {/ G C°°(e) \pzAf) < 00 for all Z G ^(c), d G N} 
be the Schwartz space of e, where 

(4.7) Pz,,(/) = sup|Z/(r)|(l + ||T||)'^. 

TGc 

Extend to Nj by setting 

X'yiuexp Z) = x-riu), for all u eU, Z £ f. 

Given / G C°°(A^f\A^; X7), define / G C°°(e) by f{Z) = /(expZ), for ah Z G e. Conversely, given 
/ G C°°(e), set /(nexpZ) = X'y{''^)f{Z) for all n G N^, Z £ z. Clearly these two maps are inverse 
of each other, and if we set 

yiNf\N; x^) = {/ e C°°(iVf\iV; X7) I / e ^(01, 
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then 



(4.8) ^^^^^(iVf\iV;x^). 

In the rest of this article we will frequently use this realization of .5^^^ (implicitly). 

Remarks 4.4. (a) Here, and in the sequel, for a smooth representation a of a closed subgroup B of 
a Lie group A, 

C°°(S\^; a) = {f e C^{A; a) \ f{ba) = a{b)f{a), for all 6 G 5 and a e A} 

denotes space of the representation of A smoothly induced from a. Later, we will also need to 
consider the space C^{B\A;a) consisting of those elements in C°°{B\A;a) with compact support 
modulo B. 

(b) In the literature, the Lie algebras u, n are frequently denoted by n2 and rii, respectively. When 
that is the case, the Lie algebra is often denoted ni.5. 

(c) When i is non-Archimedian, we set ^(c) to be the space of all the locally constant, compactly 
supported functions on e (known as the Bruhat-Schwartz space on e). With this definition equation 
(j4.8p remains valid for i non-Archimedian. 

Let ^7(0) be the universal enveloping algebra of g. Given / G C°°(G), Z E U{q) and d G N, we 
set 

qz,dif) = sup \Rzf{g)\\\g\f, 

geG 

where Rz acts on C°°{G) via the right regular representation of U{q). Let 

^(G) = {/ G C^{G) I qxAf) < for ah X G U{s), d G N} . 

It is easy to check that ^[G) is a Frechet space and that qz^d is a semi-norm on 'rf{G) for all 
Z G U{q), d G N. We call ^{G) the space of rapidly decreasing functions on G. 

We now define certain space of rapidly decreasing functions on N\G. Given / G C°°{N\G; -S^xi)^ 
Zi G ?7(g), Z2 G i^(e), di, ^2 G N, set 

(4.9) qz,,ZM.{f)= sup \Z2(Rzrf{mk)){T)\{\-r\\nf^\\rnf\ 

keK,meM,Tec 

Then we define 

<^(iV\G;^;,J = {/ G C7~(iV\G;^;,J | qz„ZM2if) < for all qz„ZM2 as in gj])}. 
Observe that 

qZi,Z2,di,d2if) = sup pz2,d2iRzJ{mk))\\mf\ 

k&K,iri&M 

where Pz2,d2 is as in (|4.7p . In general, given a smooth representation of N , (r, J^), and a seminorm 
p on we set 

(4.10) qzApif)= sup p{Rzf{mkmmf, 

keK,meM 

and define 

^{N\G; ^) = {/ G G°°{N\G; ^) | qz,dAf) < ^ for ah qzAP as in dUni)}. 

Recall that there exists a covering M^^ Mx such that (Px-y^'^x-y) extends to a representation 
of M^^ X N. Using this extension, we define a natural action of My.^ on ^[N\G] ^x-y) 

(4.11) m- f{g)= py^{m)f{m-^g), for all m G M^^, 5 G G. 

Here rh is the image of m under the map M^_^ Mx- 
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Now, given / G C~(A^f\G; X7), Z G C/(g) and d G N, we set 

qz,d{f) = sup |i?z/((expT)mA;)[||(expT)mA;||'^, 

keK,meM,Tec 

and define 

^(iVf\G; X7) = {/ e C°^{Nf\G; x^) \ qzAf) < 00 for all Z G U{g),d G N}. 
Lemma 4.5. As G-modules, we have '^{Nf\G;X'i) - '^{^\G;-^x-i)- 

Proof. Given / G ^(Ar\G;^;^J, define / G C°°(iVf\G; X7) by f{g) = f{g){Q). We claim that 
/ G ^(7Vf\G;x7)- Effectively, given Z G C/(g), d G N, 

= sup [i2z/((expT)mA;)|||(expT)mA;||'^ 

= sup |i?z/(nmA:)(0)|||(expT)mA;f 

< sup |i?z/("iA;)(r)[||m/cf llexpTf 

< sup Cs\Rzf{mk){T)\\\mkf{l + \\T\\f'-'' 

keK,m£M,T€e 

< C-iqz,lA2rd{f) < 00, 

where we have used equation (j4.4p in the next to last equality. 

Now given / G <^(iVf\G; X7), set f{g){T) = /(exp(r) g), for ah 5 G G, T G e. We claim that 
/ G ^(iV\G;^;^J. Effectively, given Zi G C/(g), Z2 G ^(e), di, ^2 G N, we have that 



qz^,ZM2if) = sup \Z2{RzJ{mk)){T)\\\mkf'{l + \\T\ 

k£K,meM,Tee 



d2 

d2 



= sup \{RAd{mk)--^Z2zJ{ew{T)mk)\\\mk\\ ^{1 + \\T 

keK,meM,Tet 

where we have identified Z2 with an element in C/'(g) for some /. Let {Zi, . . . , Zs} be a basis of 
[/'(g). Then 

s 

Ad(mA;)~^Z2 = aj{mk)Zj, 
j=i 

for some functions aj. Since (Ad, [/'(g)) is finite dimensional, it is of moderate growth, and hence 
there exists constants Ci, di > such that \aj{mk)\ < C/||mA:||'^', for all G -fC, m G M. From this, 
equations (|4.3p and (|4.6p . we have 

s 

qZuZ2,dud2if) < CiY, sup |i?^,^^/(exp(r)mA:)|||mA;f^+'^'(l+||r|j)'='^ 

~[ k&K,m€M,Tec ^ 

^ S^iZ \Rz,zJi^MT)mk)\\\mkf^+''^\\expT\f^ 
vkiZ \Rz^zJi^MT)nik)\\\eMT)mkf^+''^+^''- 



- 1ZjZi,di+di+2d2(f^ < 

1 7 = 1 



Finally, we easily check that f = f and f = f. □ 
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Given / G '^{G), define a function /^^ G C~(iVf\G;x7) by 

fx-yia) = f{ng)x-i{ny^dn. 
JNf 

Lemma 4.6. The map f /^^ defines a surjective G -intertwining map from '^{G) to the space 
'^{Nf\G;x^)- 

Proof. We will first show that /^^ G ^(iVf\G; x-y)- Given Z G U{q), d G N, we have that 
Qz,difx-y) = sup |iiz/x-,((expr)mA;)|||(expr)mA;||'^ 



Now we know that for all di G N, |i?2/(n(expZ)m/c)|||n(expZ)mA;||°'i < qz,dx{f)- Hence, from 



< sup / |ii^/^^(n(exp Z)mfc)| ||(exp r)m^ 

k,m,T J Ni 

now that for all c 
Corollary [131 

qzAfx-y) ^ sup qz,di{f) \\fi{e-xp Z)mk\\~'^^ dn\\{expT)mk\f 

keK,meM,Tec JNf 

f d 

< sup qz,di{f) / (Coll^llll expr||||m||) ''o (|| exp r|| ||m|[) dh 

k&K,m&M,Tee J N, 



= sup qz,dAf)Co " (II expr||||m||) ''o / ||n|| ^^n. 

keK,meM,Tee JNf 

But it is clear that, if di ^ 0, then the right hand side of the above equation is finite. 
Now we will show that the map is surjective. Fix a function (j) G C^{N^) such that 



/ Hn)x^ih) ^dh = l. 
JNf 



Given / G ^{Nj\G;X'r)i l^t h G C°°(G) be given by h{h{exp Z)mk) = (j){n) f {nmk) . Then it is 
clear that h G ^(G) and h^^{nmk) = f{nmk) for all k & K, m € M and Z G e. From all this we 
conclude that the map / i-^ /^^ is surjective. □ 

Remark 4.7. When i is non-Archimedian, we set ^{G) to be the space of locally constant, compactly 
supported functions, with analogous definitions for '^(iV\G; ^5^^^ ) and 'rf{Nj\G;x-y)- With these 
definitions, it is straightforward to check that all the results in this section remain valid in the 
non-Archimedian case. 

4.4. Realizing a generalized Whittaker model on N\G. In this section, we give the promised 
realization of the space of generalized Whittaker models for a Casselman-Wallach representation of 
G. But before stating this result, we need the following technical lemma. 

Lemma 4.8. Let (vr,"^) be a Casselman-Wallach representation ofG. Then 

{y')Nf,x-y ^ Wh^(7r). 

Proof. Given A G Wh^(7r), set X{v) = A(u)(0), for v T . Then it is clear that A G (r')^f'^^. On 
the other hand, given A G (r')^f'^^, set '\{v){T) = A(7r(expr)v) for u G T G e. Then it is clear 
that \{v) G C°°(e), but we claim that it is actually in S^{t). To see this, observe that if G e, then 

(4.12) X{dTT{R)v){T) = DrX{v){T), for ah T G e, 

where G i^(e) represents the derivative in the direction R. On the other hand, if S" G f, then 

(4.13) \{dTr{S)v){T) = dxy{n{S,T))X{v){T), for ah T G e, 

17 



where dx^y is the hnear functional given by the derivative of the character x-y- But now, since (vr, Y) 
is a representation of moderate growth, we can find a constant d > such that for ah v (z 'f, there 
exists Cx^y > such that 

(4.14) |A(t;)(r)| = |A(^(expr)t;)| < Cx,v\\expT\f < C-iCxA^ + WTWf^" . for all Tee. 

Here C3 and r are as in equation (j4.4p . Observe that, although the constant Cx,v depends on v (and 
on A), d is independent of the element v ^ 'f chosen. Therefore, from equations (|4.12p . (j4.13p and 
(j4.14p for aWv^'V the function \{v) is such that if we take any partial derivatives, and multiply by 
any polynomial on e, it still has growth controlled by a polynomial of degree 2dr. But this implies 
that \{v) € ^(e), as we wanted to show. 

To finish the proof we just have to show that A = A, and A = A, but this follow easily from the 
definitions. □ 

Given an s[2-triple 7 = {X, H, Y} C 0, set 

(4.15) ^ = {-X,H,-Y}. 

Then, it is immediate to check that 7 is also an s[2-triple and that x-y = X-y^ where x^ is the 
character of U given by X-yi'^) = X'y{'^)~^ for all u E [/. 

Proposition 4.9. Let (vr, 1^) he an irreducible Casselman-Wallach representation of G, and let 
'^{N\G;S^-^^)g,-k denote the maximal -isotypic quotient of ^{N\G;^-^^). Then, as an M-^^ x G- 
module, 

^{N\G-yx-)G,^^Wy{f)(i?)T^, 

where n is the contragredient Casselman-Wallach representation of tt and Wy{n) is the continuous 
dual 0/ Wh..y(7r). Here (8) stands for the completed projective tensor product (of two locally convex 
topological spaces). 

Proof By Lemma we have '^{NXG;^^-) ^ '^{N^\G;Xy)- Let A G HomG('^(iVf\G; X7), ■>^), 
and let (vr, 'f) be the representation contragredient to (vr, 'f). Then we may think of A as an element 
in a space of '^'-valued distributions. Hence, according to [14] there exists fix G (^')^f'^T such that 
if/GC7~(iVf\G;x7),then 



Kf)iv) = / fig)l^xi7Tig)v) dg, for v € Y. 

JNf\G 

On the other hand, given G (-^')^f.X7 and / G 'tf{N^\G;Xy), there exists h G "^(G) such that 
/ = hy.- and for all t> G 

/ h{9)li-\{T^{9)v)dg = / h{hg)iix{T^{n)TT{g)v) dhdg 



Hng)X'y{n) dn nx{'K{g)v) dg 

Nf\G JNf 

h{f^9)Xy{n)~^ dh fix{TT{g)v) dg 

Nf\G JNf 

Kii9)l^\{'^{a)v)dg 

Nf\G 

f{9)lJ'\i'^{9)v) dg. 

Nf\G 
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Therefore, if we set 

Am(/)W=/ f{g)i^x{T^{g)v)dg, ioivef, 

JN^\G 

then G Hom.G{'^{N^\G]X'y)^^')- Method of |361 Section 3] impHes that A^(/) extends to a 
continuous hnear functional on Y', i.e., \fj,{f) is actuahy an element of 'f for all / € 'jif{Nj\G;X'y)j 
or equivalently G IlomG{'Tf{N^\G;x^)i^)- Now it is easy to check that A^^ = A and fix^ = /i. 
Therefore 

Finally, according to Lemma [4.81 t^n^,x-i ~ ^-yi^) proposition follows. □ 

Remarks 4.10. (a) In the Archimedian setting whenever we take the tensor product of two locally 
convex topological vector spaces, we always mean the completed projective tensor product. Observe 
that if (vr, 'V) is a Casselman-Wallach representation, then 'f is nuclear, and hence the completed 
projective tensor product of Y with any locally convex topological vector space is equivalent to the 
completed injective tensor product [5j. 

(b) Proposition 14.91 also holds for k non- Archimedian, if we replace in its statement Casselman- 
Wallach representations by smooth, finitely generated, admissible representations, and completed 
projective tensor product by algebraic tensor product. The proof for this case follows the same line, 
but it is more straightforward. 



5. Dual pairs and theta lifting of nilpotent orbits 

5.1. Reductive dual pairs of type I. Let {V,B) be an e-Hermitian module, and let {V,B) be 
an e-Hermitian module such that ee = —1. Let G = G{V) and G = G(V) be its associated 
isometry groups. We will use the e-Hermitian form B to identify the i- vector spaces V 00^* and 
RomD{V,V). Given T G RomD{V,V), define T* G RomD{V,V) by 

B{Tv, v) = B{v, T*v) for allvGV,ve V. 

Define similarly the * map from Homj:)(y,y) to ^omoiV^V). Note that since ee = —1, we have 
T** = -T for T G Homz)(F, V). 

We now define a symplectic form (•, •) on Hom£)(y, V) by setting 

(5.1) (T, S) = Ti{T*S) for aU T, 5 G Homz)(y, V), 

where Tr(T*5) is the trace of T*S as a i-linear transformation. Let 



Sp(HomD(l^,y)) = {g(^ GL(HomB(y,y),i) 



{g-T,g-S) = {T,S) 
for ah T, 5 G mmD{V,V) 

Then there is a natural map G x G — > Sp(Hom£)(y, V)) given by 

{g, g)-T = gTg-^ for aU T G Hom(y, V), g e G, ~g ^ G. 

We will use this map to identify G and G with subgroups of Sp(Hom£)(y, V))- These two subgroups 
are mutual centralizers of each other, and form an example of a reductive dual pair of type I. See 
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5.2. Moment maps and lifting of nilpotent orbits. Given T e HouidIV^V), it is clear that 
T*T E g and TT* S g. Following |16^ |3] we define the moment maps to be 



ip : RomD{V,V) — > g 

T ^ T*T 

and 

(f :Rom.D{V,V) — g 

T ^ TT*. 

It is also clear that (p{T) is nilpotent if and only if (^(T) is nilpotent. 

As in the introduction, let Max Hom(y, V) be the set of full rank elements in Hom(y^, V) . Without 
any loss of generality, we assume that dimF < dimF, and elements of MaxHom(y, y) are then 
represented by injective maps from V to V. Recall also our standing assumption on the nilpotent 
orbit O C g: 

(5.2) ^~^{0) n MaxHom(y, V) + 0. 

We first prove a result on dual pairs in the stable range, which says that in this case any (not 
just nilpotent) orbit satisfies the stated assumption. Recall that the dual pair (G, G) is in the stable 
range, with G the smaller member, if there is a polarization V = E (BU (B F, where E, F are totally 
isotropic complementary subspaces with diuiE = dimF = dimV. 

Lemma 5.1. Assume that the dual pair {G,G) is in the stable range with G the smaller member. 
Given X & q, there exists an injective map T £ Hom(y, V) such that T*T = X . 



Proof. Fix a linear isomorphism Tg, : V ^ E and define a linear map Tp : V ^ F hy Tp = 
(Tty^X/2. Then, if we set T = Tp + Tp, we have that 

T*T = {T*p + T*p){Tp + Tp) = T*pTp + T*pTp = X/2 + X/2 = X. 

Note that we have used the fact that Tp = X*{Tp)^^/2 = XT^^ /2. Now, since Tp is injective, we 
conclude that T is injective and T*T = X as we wanted to show. □ 

Remark 5.2. The correspondence of nilpotent orbits for dual pairs in the stable range is well under- 
stood. See [3l|27], and [24| (for i^c-nilpotent orbits). This is also covered by the next proposition, 
in view of Lemma 15.11 

Proposition 5.3. Let O C Q be a nilpotent orbit corresponding to the admissible e-Hermitian 
Young tableau T = (d^, {V'^ , B^)), where = [4% • • • , ]• Given X e O, letT € Hom(y, V) be an 
injective map such that T*T = X . Then the orbit of the nilpotent element X := TT* € g corresponds 
to the unique equivalence class of admissible e-Hermitian Young tableaux F = {d^ , (V^ , B^)), where 

• = [(ti + 1)*S • • • ; (i« + l)*'i f^ith s = dimV — dimV — Yl'^j=i h> 'namely f is obtained 

by adding a column of length dimV — dunV to T; 
. {v!/^'-\b]:'^^^) - {yf:XXfiA), for allj = l,..., l; 



(Vq'^jBq'^) is an e-Hermitian module of dimension s. 



Proof. Let 7 = {X,H,Y} be an s[2-triple containing X. Define V^'^^ as in Section [3.11 and set 
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(At this point, 7 has no meaning.) Since T is injective, we have a direct sum decomposition: 

I 

T(y) = 0r(y^'*.), 

and for each j, we have 



i=l i=l 

Observe that for ell v, w , 

B{Tv,Tw) = B{v,T*Tw) = B{v,Xw). 

This imphes that T{Vt''^')±T{V'^''^J) for i / j. We also note the fohowing: iiA,BcV and A, X{B) 
form a perfect pairing under B, then T{A),T{B) form a perfect pairing under B. This imphes 
that, for a fixed j, ^-t*+2j is non-degenerate, and V^J'*''"'' is orthogonal to 0*Li Vll^^^^f^j- 

Therefore there exists v'^f^'^^ such that 



r(yT'*0 e Kjf = 0-Lo^-?+2* is non-degenerate; and 
V2f'^^ ±T{V^^^^), for ah i / j. 



By an inductive argument, we may thus find V2^t*^^^ for 1 < j < / such that 

• tlj =: ©iLo^-t*^^! is non-degenerate; and 

• (B^j^iUj is an orthogonal direct sum. 

Let V^'^ be the orthogonal complement of ©^-^j^i/j. Then, we have a decomposition 

where ffl denotes an orthogonal direct sum. 

Let Vi = (BjVi''^''^ , and define an element H (easily checked) by setting Hv = iv for all 
V G Vi. Note that for i < tj - 1, T*\.j,t^+i : l7.'^'*^+^ ^ V^_^l' is an isomorphism and TT* = X : 

V^' ^ ^i+i ■ particular, X is nilpotent. Since = 2X and H is semisimple, there 

exists an element Y such that {X,H,Y} is an s[2-triple. See [2,, Section 3.3]. From all this it 
is clear that if T = {dF , iV^ ,B^)) is the e-Hermitian Young tableau associated to the orbit of X, 
then dF is obtained by adding a column to dF . Finally observe that X^^ = TX^^^^T* , and so 

x-'^\-,,t^+i = (r-i)*x-(*^-i)r-i[ 

liv,we V;]'*'^\ then 



Bl^'^\v,w) = B{X-'^i,w) 



(5.3) 



B{X-^^^-^^T-^d,T-^w) 
B^:%{T~\T-'w), 



which implies that {VtY'^\ B^f'^'^) ^ {v[:XbI'\), for all j = 1, . . . , □ 
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Let O C be a nilpotent orbit (satisfying (j5.2p ). which corresponds to an e-Hermitian Young 
tableau T = {(f , (V^ , B^)). We define the theta lift of O, 6(0) C g, to be the nilpotent orbit 
corresponding to the e-Hermitian Young tableau T = {dF , {V^,B^)) specified in Proposition! 



Remark 5.4. It can be shown that, for the orbit O under consideration, ip{ip ^{O)) = @{0). Cf. 
^16^ Lemma 4.3]. Therefore, our definition of the theta lift of O agrees with the usual one. 

5.3. Lifting of s[2-triples. In this section we will introduce the concept of lifting of s[2-triples. We 
start with the following 

Definition 5.5. Let 7 = {X,H,Y} C Q and 7 = {X,H,Y} C g be two s[2-triples of type O and 
6, respectively. We say that T G Hom(l^, V) lifts 7 to 7 if T*T = X, TT* = X and T{Vj) C Vj+i 
for all j. 

Set 

(5.4) O^,^ = {T G Hom(F, V) \ T lifts 7 to 7}. 

We prove the following elementary 

Lemma 5.6. Let 7 = {X, H, Y} C and 7 = {X, H, Y} C g be two 5l2-triples. Let T G O^^^j, and 
denote 

Then Tj is injective for j < 0, and surjective for j > 0. 

Proof. We have X\vj = T*T|v^. = T^,_2?j'. Since all highest weight vectors of a finite dimensional 



s[2 representation have non-negative weights, we know X\v^ is injective for j < 0. This implies the 
first assertion. 

For the second assertion, we first note that Tf : V-i-i 1— )• V-^i. We have X|,> = TT*\,y = 
T-jT* . By the same reasoning, we know that T* is injective for j > 0. The second assertion 
follows. □ 

From now on, we assume that O satisfies (|5.2p . as in the introduction. We let 

(5.5) Oi^l^ = O^,^ n MaxHom(y, V). 

Lemma 5.7. Let 7 = {X,H,Y} C g and 7 = {X,H,Y} C Q be two 5l2-triples of type O and 
6(0), respectively. Then 0}f^^ / and it is a single Mx x Mj^-orbit. 

Proof From the proof of Proposition [SiH we know Oi^l"" / 0. Now let T G O^l'' and set 

Tf'*' = T\ 7,t, for ah I < j < I, -tj + I < i < tj - 1. 

We claim that T^^'*^ defines a linear isomorphism between V[''^^ and V^^i'^^ for all 1 < j < / 
and —tj + 1 < i < tj — 1. Since TX = XT, T(Vi) C V^+i and T is injective, we clearly have 
T(y^^'^\) C V^J'*''^^. By Lemma [5.6| Tj is surjective for all j > 0. Thus any element of V^^'^'^^ is of 
the form T{v) for some v G Vtj-i- But then XT{v) = TX{v) = and since T is injective, we must 
have X{v) = 0. Thus tI% : V^Z'i ^ yj' ^ + is a linear isomorphism. By applying an appropriate 

power of X, we may conclude that T-'^ is a linear isomorphism between V- ' ^ and V^_^/ for all 
1 <j <l and -tj + 1 <i < tj - 1. 
We set 

^7'*^' = ^L.-*. and 17'*^+^ = X| ,,,+1. 

t i 

22 



Then it is clear that X]'^' = (T^fia)*^^'*' and x/'*'"^^ = T^]^( {T^JLiT ■ From this we conclude 
that T is completely determined by the maps T^^'l^, for j = 1, . . . , / , but now, from equation (15.3 
all the T^.'j!iS are isometries. From this and equation (I3.7p it follows immediately that O^^^ is 



IS a 

single Mx x M^-orbit. □ 

Let 7 = {X, i?, y} C and 7 = {X, ^, y} C g' be two s^-triples. Given T G we define a 

map (/>T : Mj^ — > Mx by 

(5.6) ^T{rh)v = {T'^'^'Y^mT^'^'v, 

for all 771 G Af^, and v (zV- ' ^ , where 1 < j < / and — + 1 < i < tj — 1. Note that as elements of 
Ilom(y,V), we have 

rhT = T(j)'T{rh), 

for all rh € Af^. 

5.4. Isomorphism of 0_i © g-i with a symplectic subspace ^7,7 of ILom.{V,V). Let 7 = 
{X, H, Y} and 7 = {X , H, Y} be two s[2-triples of type O and Q{0), respectively. From the proof 
of Proposition 15. 3| we may assume that 

r r+1 

V = ^Vk and V= Vk, 

k=—r k=—r—l 

for some r. We will set 

r 

(5.7) W^,^ = Hom(Vfc, I4) C Hom(y, V). 

k=~r 

Then it is clear that the restriction of the form (•, •), defined in Section [STTl to VF^,^ is non-degenerate. 
Now fix T G 0^1^, and define 

..^N Jt ■■ 0-1 ©0-1 — > W^,7, 

^ ' {R,R) ^ TR + RT. 

Obviously, we have Jx{R, R){Vk) Q Vk for all k. 

Lemma 5.8. Jt defines an isomorphism between g_i © 0-i and W^,^. Furthermore, 

{Jt{Ri,Ri),Jt{R2,R2)) = -ti-i{Ri,R2) + ~^-i{RuR2), 

for all Ri, R2 G 0-i; Ri, R2 G 0-i- Here the bilinear form k on q is as in Section \2.^ and likewise 
for k. 

Proof. First observe that if R2 G 0-i, then 

{TRi,TR2) = Tt{{TRi)*TR2) = Tt{RIT*TR2) 
= - Tt{RiXR2) = Ti{R2XRi) 

= -K^i{Ri,R2). 

Similarly {RiT,R2T) = K;_i(i?i, i?2)- On the other hand, if i? G 0-i, R G 0-i, then 

{TR, RT) = Tt{R*T*RT) = - Tv{RT*RT) 

and 

{RT,TR) = Tt{T*R*TR) = - Tr{T*RTR). 
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Therefore, {RT,TR) = 0, since (•, •) is symplectic. We have thus shown that 

{JTiRl,Rl), Jt{R2,R2)) = i?2) + i?2), 

for all R2 G 0-1, R2 G 0-i- But now, since k_i and k_i are non-degenerate, we conclude 
that Jt is injective. 

Now observe that, for all Sk G Hom(Vfc, Vfc_i), Sk — £ On the other hand, we can write 
S £ Q-i as the direct sum Yll:=-r '^k, where Sk € Hom(Vfc, Vk-i). Then we have that SI = —S^k+i 
for all k, and hence, the map S S — S* defines a linear isomorphism between 0fe<o Hom(Vfc, Vk-i) 
and g_i. From this, we conclude that 



dimg_ 



-r+l 

dim Vfc ■ dim Vfc.i 

k=0 

-r+l 

dimVfc-i ■ dimVfc_i 

k=0 



k=-l 



dim Vk • dim Vk ■ 



Here we have used the fact that dimV^ = dimVfc_i for A: < 0, in view of the linear isomorphism 



Tlk ■■ Vk-i 



Vk- Similarly, 



dim0_ 



dim Vk • dim V^- 

fc=o 

—r 

y^ dim Vk ■ dim Vk 

k=0 

r 

dim Vk • dim Vk ■ 



k=o 



Therefore, 



dimg^i + dim0_i 



y^ dim Vfc • dim Vk + dim Vk • dim 14 

k=-l k=0 
r 

dimHom(Vfc, Vfc) = dimW-y 



7,7' 



and hence, Jt must be a bijection. 

6. Generalized Whittaker models and Howe correspondence 



□ 



6.1. The smooth oscillator-Heisenberg representation associated to ^7,7. Let H^^^ be the 
Heisenberg group associated to the symplectic space ^7^7. That is, H^^^ = W-y^^ x i, where {0} x i 
is central, and {R,0){S,0) = {R+ S, {R,S)/2). Let Mp(VF7^7) be the metaplectic group associated 
to W-y^^, and (r^f^^y, Jif^^^y) be the smooth oscillator-Heisenberg representation of Mp{W'y^y) x i^7,7 
associated to the character ip. 

Recall that we have fixed T G O^?^, and we have an isomorphism Jt from 0_i©g_i to 1^7,7, given 
in equation (jS.Sp . Define a new invariant bilinear form k' on g by setting k'{R,S) = —k{R,S), for 
all R, S £ Q. This results in a new symplectic structure on g_i, given by k'_^{R, S) = —K^i{R, S), 
for all R, S £ g_i, and hence a new Heisenberg group H'^. With this new symplectic structure, 
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•^rlg-i is a morphism of symplectic spaces, and hence, we may extend Jt to an injective morphism 
of groups Jt : H'^ — > -^^7,7- Observe that we can proceed similarly for g_i, but for this space the 
modification of the symplectic structure is unnecessary. We obtain in this way a map 

(6.1) Jt-.H'^xH;^ — > H^^^y. 

As in Section [3.31 we let a'^ : N — > H'^ be the map induced by 7 and the bilinear form k', that 

is 

a^^(expi?expZ) = {R,k'{X,Z)), for all R G g_i, Z G u. 
Similarly, define : — > H^y. By composing with the map Jt, we have a group homomorphism 

(6.2) ot = aT,7,7 ■ N X N — > H^^^ 

given by aT{n,h) = JT{a'^f{n),a^{h)). 

Now observe that for all m G Mj^, R G g_i, 

(6.3) JT{M{m)R) = mRm'^T = mRT(f)T{my^. 

On the other hand, since (j)T '■ M-^ — > Mx is surjective, then for any m G Mx we can find rh G M-^ 
such that (pTi'rn) = m. Moreover, for all R G 0_i, 

(6.4) JT{Ad{(t>T{m))R) = T4>{m)R(j){my^ = mTR(l)T{mY^. 

Using equations (I6.3p and (|6.4p . we define an action of Mx x Mj^ on ^^7,7, by 

'ihS(pT{-m)~^ if 5" G Jt(0-i) 
S if5GJT(0-i), 



m ■ S 



if rh G Mj^ , and 



r s if5GjT(0-i) 

\ mSM^r^ if 5 G Jr(g-i) 



if m = (j)T{rh) G Mx, for some rh G M-^. Observe that in particular, for all rh G M^, and all 
S G ^^7^7, we have 

(6.5) {m(j)T{rh)) ■ S = rhS(l)T{m)~^ . 
Using this action we extend the map ot to a group homomorphism 

(6.6) aT ■■ M^^N X M^^N — > Mp(W7,7) x H^^^. 
By pulling back, this yields a representation 

(6.7) r^^ := r7,7 o ut 

of M^_^N X M^;yN on =^,7. Observe that for all Z G u, u G =^,7, 

r^;^(exp Z, e)t; = T-y^x^{<d, k {X, Z))v 

= iP{k'{X,Z))v =iP{-k{X,Z))v 
= X7(exp Z)v = X7(exp Z)v, 
where e is the identity element in M^-N. Similarly, for all Z G u, w G .^,7, 

r^^(e, exp Z)v = x^{ex.p Z)v, 

where e is the identity element in M^^N. Using this, and the fact that Jt '■ 0-i © 0-i — )• ^^7,7 is 
an isomorphism, we obtain the following result. 
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Lemma 6.1. As M^^N x M^-N -modules, we have 
where r^- = r^,^ o ax- 

6.2. The main result and the key proposition. We recall some notation from Section [H Let 
iy,B) and iV^B) be an e-Hermitian and an e-Hermitian Z)-module, respectively, with ee = — 1. 
Let G{V) be the corresponding isometry groups and g = 0(1^), = QiV) their Lie algebras. 

Then G{V)) form a dual pair in Sp(Hom(y, V)) in the sense of Howe [7]. 

Let Mp(Hom(y, y)) be the metaplectic group. This is the unique topological central extension 
of Sp(Hom(y, V)) by {±1} such that it splits if k is C, and it does not split otherwise. Denote the 
pullbacks of G{V), G{V) in Mp(Hom(y,y)) by G and G, respectively Then (G,(5) form a dual 
pair in Mp(Hom(y, F)). Let be the smooth oscillator representation of Mp(Hom(y, y)) 

associated to the character ij:. It is well-known that after tensoring with a genuine character of 
G X G, {ijj, '3^) yields a representation of G{V) x G{V), except when one of them, say V , is an odd 
dimensional quadratic space and i ^ C. In this exceptional case, G is the metaplectic cover of G(y), 
and we shall only consider genuine representations of G. For all other cases, representations of GiV) 
are identified with genuine representations of G (by tensoring with a fixed genuine character of G). 
With this convention/caveat in mind and for the rest of this article, we will make little distinction 
between G and G{V), and loosely speak of genuine representations of G. Likewise for G and Giy). 
This convention will lead to significant savings in notation and no confusion is expected for the 
expert reader. 

We state the main result of this article, in a more concrete form than Theorem 11.11 of the intro- 
duction. 

Theorem 6.2. Let {G,G) he a reductive dual pair, and let (w, ^) he the smooth oscillator repre- 
sentation associated to the dual pair (G, G) and to the character ip of k. Let (vr, 1^) he a smooth irre- 
ducible genuine representation ofG. Let O C Q he a nilpotent G-orhit in the image o/MaxHom(y, V) 
under the moment map (p and let O = 0(O) G Q be the corresponding nilpotent G -orbit. Then 

Whe(o)(e(7r)) ^Who(7r), 

where vr is the contragredient representation o/vr. More precisely, let 7 and 7 be two 5\2-triples of 
type O and @{0), respectively. Then, given any T € O}^'^^, there exists an isomorphism 

$T : Wh^(^) — > Wh^(e(7r)), 

such that 

m^>T(A) = ^>t((/'t("T')A) for all m G M^-, A G Wh^(7r), 

where cpT '■ My.- ^'^ equation ( t5. 6]) . 

The key to our proof of Theorem 16.21 is the following result. See Section 3] for the unexplained 
notation. 

Proposition 6.3. Let 7 C and 7 C 6e two 5i2-triples of type O and 0(0), respectively. Then, 
given any T G O^^^, there exists a G x My^.N -intertwining isomorphism 

(6.8) *T : ^u,x, "^(.NXG; M,^^), 

where M^^N x M^-N acts on J^^^y via the representation t^- defined in equation Ii6. 7j) . and the 

action of G x M^-N on 'if(N\G;Jif-y^^) is defined in the following way: given f G '^{N\G; J^^^/) 
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and g ^ G, 

(6.9) {g-f){9) = figg) fordlg'eG, 

(6.10) {n-f){g) = Tl^{h)f{g) for all fi e N 

(6.11) {m-f){g) = Tl,^{{Mrn),m))f{Mrnr^9) for all rh e M^- . 

Before starting with the proof of this proposition, let us show how it imphes Theorem 16.21 
Proof (of Theorem \6. ^J . By Lemma l6.ll we have 

^{N\G; ^^,^) ^ ^{N\G; ®^x,) = '€{N\G- ® 

Now, given / G '^(iV\G;^^^) and v G we identify / (g) u with an element of ^{N\G\ M'-y^^y) 

via the above isomorphism. With this identification, we may rewrite equation (j6.1ip as 

rh- if (g) v){g) = T^^;y{(l)T{rh))f{(pT{my^g) r^;^(m)'y 

= {(pTirh) ■ f){g) (g) r^;^(m)f , 

for all fh G Af^,. See equation (j4.1ip . 

By Propositions 16.3 1 and 14.91 we have the induced isomorphisms: 

^c/,x,),{G,.) = nN\G; ^x.)(G,.) ^ ^x, = ^ ® W,{r:) ® , 

where M^, acts on W.y{'k) through the map (pT '■ ~^ ^x-y- other hand, we have (from 

the definition of G(7r)) that 

Therefore, there exists an isomorphism 

$T : Wh^(^) Wh^(e(^)), 

such that 

m$T(A) = ^Ti4>T{rhX)) for ah rh G M^-, A G Wh^(7r). 

□ 

In the rest of this section we will closely examine the space _ . In fact we will prove a 
refinement of Proposition [6]3] in which we give an explicit isomorphism ^j- in equation (|6.8p . This is 
Proposition [631 As we have already seen, our main result follows immediately from this proposition. 

6.3. Strategy for the key proposition. Let 7 = {X,H,Y} C g and 7 = {X,H,Y} C 7 be two 
s[2-triples of type O and @{0), respectively. From the proof of Proposition 15. 3| we may assume 
that 

r r+1 

(6.12) v = ^Vk, and ^ = Vk, 

k=—r k=—r—l 

for some r > 0. Now fix Ty^y G Oi^^"^, that is, fix an injective element T^^^ G Hom(y, V), such that 
T*-Ty^y = X, Ty^yT* - = X and Ty^^iVk) C Vfc+i for all — r < k < r. Then we have a decomposition 

(6.13) Ty^y = eL-r^fc, 

with Tfc G Hom(Vfc, Vfc_|_i), and T/. is an isomorphism for A; > 0. C.f. Lemma 15.61 Figuratively 

V-r e V-r+1 • • • e Vr-1 e Vr 

\/T-r \jT_r+l • • • \tTr~l\iTr 

V-r-1 e V-r © V-r+1 © • • • © Vr-1 © K © K+l- 
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We will use the gradings of V and V associated to H and H, respectively, to describe the 
so-called mixed model of the smooth oscillator representation (uj,^) associated to the dual pair 
(G,G). Using this model we will give a convenient description of the space of covariants "SC-r 
leading to Proposition 16.41 But before we outline the strategy, let us fix some notation. For / < r 
and m < r + 1, set 

I m 

(6.14) = Vfc, and V(„) = 14. 

k=—l k=—m 

Let = {g ^ G\g ■ V = V ior all V ^ ^(0^' '^'^i*^'^ isomorphic with G(V(;)). Let (m)) ^(z),(m)) 
be the smooth oscillator representation associated to the dual pair and the character 

ip of k. Set y{i)^m = o5^(Hom(V(;), V^)), the Schwartz space of Hom(V(;), V^). Similarly, set 
^^^) = y{YLom{VuV(^rn))) and = ^(Hom(yz, V^)). (Note that (^\ can be identified 

with the Schwartz space of a Lagrangian subspace of Hom(V(;), V(^)), through the Schrodinger 
model.) Now observe that 

Hom(y(^), %+!)) = Hom(y(^), Vr+i) Hom(y(^), y_^_i) Hom(y(^), %)) 

= (Hom(V(r), V;+i) Hom(y_^, %))) (Hom(V(^), Vl^_i) © Hom(K., V^r))) 
©Hom(y(,,_i),%)) 

and Hom(V(^), Vr+i) ©Hom(l/_r, V(r))> Hom(V(j,), V-r-i) (B^omiVr, V(r)) are totally isotropic, com- 
plementary subspaces. It then follows, from the standard theory of the oscillator representation [9], 
that 

^r),{r+l) = ^(r),r+l ^ %),(r) 

= [^{r),r+l ^-r,(r)] «> %-l),(r), ^ > 0. 

What this equation is saying is that we may interpret the space ^^^^ (^^.i) as the space of Schwartz 
class functions on Hom(V(^), 14_(_i) with values in ^^^) (-^), and the latter, in turn, can be in- 
terpreted as the space of Schwartz class functions on Hom(y_,r, V(r)) with values in 
More concretely, we describe this space in the following way: given p a seminorm on ^^^) (^), 
Z € ^(Hom(V(r), V^r+i)) (^ denotes the space of constant-coefficient differential operators), d G N 
and / G C°°(Hom(V(,),V;+i); %),(,)), set 

qz4Af)= sup p(^/(r))(i + ||r||)^ 

TGHom(y(,)K.+i) 

where ||T|| is the operator norm of T. Then 



^(r),r+l®%Ur) = {f^ C°° (Hom(y(,) , V;+i ) ; ^^,),(,) ) 



qz,d,pif) < foi^ ah d G N, 
ZG^(Hom(V(,),T4+i)), 
/> seminorm on '3^i^r),{r) 



and likewise for [^(^r),r+i^ '^-r,{r)]^'^(r-i),(r) ■ Proceeding inductively, we obtain the tensor product 
decomposition 

%),(r+l) = [^(r),r+l ^ ^-r,{r)] " " " ® [^{1),2 ® ^-1,(1)] ® ^(0),{1) 

= [^(r),r+l ^ ^-r,{r)] ® " " " ® [^{1),2 ^ ^-1,(1)] ^ ^(0),1 ® %,(0)- 

Again we may interpret the space appearing in the right hand side of this equation as the space of 
Schwartz class functions on 0fc=r,...,i[Hom(V(fc), V^+i) ©Hom(y_fc, V(fc))] ©Hom(V(o)i ^i) with values 
in ^^o),{0)- This is the mixed model of the smooth oscillator representation associated to the 7, 
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7-gradings. In the rest of this article we will make these identifications of spaces without further 
explanation. 

For simplicity, assume for the moment that O and @{0) are even orbits, that is 0_i = and 
g_i = 0. (By Section [5.4| this is equivalent to W^^^ = Hom(Vfc, 14) = 0.) In this case we 

have that ^^o),(o) = *C and we may define a continuous linear functional A(j,) € ^^J,^ (r+i) setting 

\r)if) = f{Tr,T^r, ■ ■ ■ ,Ti,T^i,Tq), 

where / G = [^(r),r+l ^ ^-r,(r)] «) • • • [^(i),2 «> ^-1,(1)] ^(0),1 ® ^o),(o)- As WC will 

see later in this section, for such an / we have that 

\r){^{r),{r+i){r>')f) = Xy^ (ri) X(^r){f) for all u G U = N , 

and 

\r){^^ir),ir+i){n)f) = X^i{n)\r){f) for ah n € C/ = iV. 
Now given / G '3^(r),(r+l)^ set 

f{r){9) = \r){^{r),{r+l)i9)f), 

for g ^ G. It is then immediate that /(^^ £ C°^{N\G; r^^^^) but we will show that actually 
/(j.) S ^{N\G] M'-y^^). (Here we note that J^^;^ is 1-dimensional since W^^^ = 0.) On the other 
hand, since and M-^- preserve the gradings on V and V , respectively, it is straightforward to 
check in this case that 

(^(r),(r+i)("t)/)(r)(s') = f{r){4>T{fri)~^g),ior all / G m G M^,, 5 G G. 

It then follows that the map / 1— ?> /(r) induces a G x Af^- A^- intertwining map 

that we will show to be an isomorphism. 

To describe the map in the general case, observe that if (t^,7, =^,7) is the smooth Heisenberg 
representation associated to 1^7,7 (as in Section l6.ip . then, following arguments similar to those 
leading to equation (|6.16p . we obtain the following tensor product decomposition: 

(6.17) J^,7 ^ ^_r,-r ^ y-r+l,-r+l ^-1,-1 O ^0),(0)- 

Again we may identify the space Mi^^^^ with the space of Schwartz class functions on Hom(y_r, V-r)® 
■ ■ ■ ® Hom(y_i, V^i) with values in ^o),(0)- Now, given / G ^r),{r+i)i set 

(6.18) f(^r){a){S-r, ■ ■ -,8-1) = {uJ(^r),ir+l)i9)f){Tr,T-r + S-r, ■ ■ ■ ,Ti,T_i + 5_i,ro), 

for all g ^ G, S^k ^ Hom(y_fc, y_fc), k = 1, . . . , r. Note that when O and @{0) are both even this 
definition agrees with the one given before. 

The following result is a refinement of Proposition 16.31 

Proposition 6.4. Let 7 C 3 and 7 C g &e two 5l2-triples of type O and Q{0), respectively. Fix 
T G O^'^^. Given f G ^r),(r+i)) define f(^r) ^ C°°{G; Jf^^^) as in equation 116.18]) . Then the map 
f f{r) induces a G x M^.N -intertwining isomorphism 

*T : {S^irUr+l))u,x, ^ ^(A^\G; ^7,7) . 

where G x M^-N acts on '^{N\G; J^-y^y) via equations i6.9\) - [6.11\) . 

As we have already shown, the existence of such a is enough to prove Theorem 16.21 Before 
proceeding to the proof of this key proposition, we introduce all the remaining notation. 

Recall that we have set G(/) = {^f G G \ g ■ v = v for all v G ^(ij}- Let Pi be the stabilizer of V^i 
in G(;). Then Pi = M^j^^Ni, where A'^; is the unipotent radical of Pi, ^{l) ~ ^ ^{l-i)^ ^-^d 
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Ml = GL(1^_;). Now observe that, if 5 G Hom(V(;_i), then 5 — 5* € n^. Hence, we can use 

the map S >—?■ S — S* to define a Lie algebra isomorphism 

(6.19) Hom(F(,_i),l/_0e3i=n,, 

where 3, = {Z e Rom{Vi,V_i) \ Z* = -Z} is central, and [T,S] = ST* - TS* for ah T, S £ 
Hom(V(;_^), y_i). In what follows we will make implicit use of this isomorphism to identify these 
two spaces. 

In a similar fashion, we define = ^-^d observe that if we set Pm to be the stabilizer 

of Vm in ) then Pm = M(„) A^m, where Nm is the unipotent radical of Pm, -W(m) = -^m x G(m-i) , 
and Mm — GL(V^). Similarly to (j6.19p . we use the map S S — S* from Hom(T^, V(^m-i)) to rim 
to define a Lie algebra isomorphism 

(6.20) Hom(y„,F(„_i))e 

where, again, 3^ = {Z G Hom(V^, V-m) \ Z* = —Z} is central, and [T, S] = S*T — T*S for all T, 
S E Hom(F„j, V"(m-i))- We will also set 

(6.21) N^i^=NnG^i), Ui = UnNi, and = n iV(,) , 

with similar definitions for N(^m) ) and U(^m) ■ Let X = X7 be the character of U associated to the 
sb-triple 7, and let xi = x\ui, X{1) = x\u^ly Similarly we define x = X7, Xm and X(m)- Finally, define 
€ End(V(/)) in the following way: for /c < / — 1 we define X(;-)|v5, = ^[vi and for I — 1 < k < I 
set X(;)|vj. = 0. Then it is clear that X^i-^ G g^^) := 0(V(/)) is a nilpotent element. Let H(^i^ = H\vf^^y 
Then H^i^ G g^^^ is semisimple, and X^;)] = 2X(-;). Therefore, there exists Y^^^ G g^^) such 

that 7(/) = , , Y(;) } is an s[2-triple. Observe that the parabolic subgroup associated to this 
5[2-triple is precisely P^^^.^ = P^ Ci G^j-^ = M^i-^N(^i^, where M(;) = M n G(/). Furthermore, observe 

that X7(;) = X(i)) and hence -^xtjq ~ ^X{i)- We make analogous definitions for X{i), 7(/), etc. 

We recall the following explicit formulas for '^(r),(r+i) IcxP +1 PJ' ^ ^ '^{T),{r+i) — ^(r),r+i ® 
^(r)^(r) and T G Hom(V(r), V;+i), 

(6.22) (^M,(.+i)(5)/)(T) = a;(,),(,)(5)[/(r5)] for ah 5 G G = G(y(,)), 

(6.23) (a;(,),(,+i)(m)/)(r) = z.(m)[/(m-ir)] for ah m G M.+i ^ GL(V;+i), 

(6.24) (^M,{.+i)(5)/)(T) = ^(,),(,)(^)[/(T)] for ah ^ G - G(V(,)), 

(6.25) (^^(,),(,+i)(expZ)/)(r) = i^{TrZTT*/2)[f{T)] for all Z G 

(6.26) (w(,),(,+i)(expi?)/)(T) = ^^(,),(,)(-i?r)[/(r)] foralli2GHom(V;+i,%)). 

Here u is a, certain character whose explicit form will not concern us. In the last equation we implic- 
itly understand that ^(^j.),{r) carries an action of the Heisenberg group associated to Hom(V(^), V(r))- 
We also observe the following: since Vr+i C y(r+i) is stable under the action of Pr+i, we may 
consider Hom(V(^), V^+i) as a G x Pr+i-module with trivial G(r) x -^r+i (C Pr+i) action. From 
equations (I6.22p - (|6.26p . it is then clear that the actions '^(r),(r+i) IgxP +1 ™ay be interpreted as 
coming from the usual vector bundle action of G x P^+i on the trivial vector bundle with base 
Hom(V(j.), I4.+1) and the fiber ^^^j ^^). (This action depends on T.) To be more precise, we may 
rewrite equations (|6.22p - (|6.26p in terms of an action ctt of G x P^+i on ^^^^ as follows: 

(6.27) HrUr+i){9)f){T) = cjT{g)f{Tg) for ah 5 G G 

(6.28) (^{r),{r+i)(p)/)(T) = aT{p)f{p'^T) forahpGP.+i, 
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where (JT{g) = ^{r),{r){9) ^or all g G, and 

o-t(p) = 4^{Tr ZTTy2)u{rh)uj(_r),{r){9p{r),{r){RT), 
for p = m^(exp i?)(exp Z), with rh € M^+i, g G G(j-), R € Hom(V^4.i, V^^)), Z £ jr+i- 
6.4. The inductive step: relating covariants of ^^r),(r+i) with (r). Given a function 

/ G ^^r),(r+l) — [=5^(r),r+l®^-r,{r)]'^^(r-l),(r)> define a new function /r e C°°(G; ^_r,-r'8'^r--l),(r)) 

by 

(6.29) fr{9){S) = [uj^rUr+i){9)f]{Tr,T^r + S), for all gGG,Se Hom(y_„y_,), 

where T^^/y = X]fc=-r^fc before. Then, from equations (|6.25p and (|6.26p . we have that for all 
/ G %),{r+i) 5 e G, 5 G Hom(y_,.,Vl^), 

(w(r),{r+i)(expZ)/)r(5r)(S') = (tj(r)_(r+i)(expZ)[w(^)^(^+i)(5r)/])(r^,r_r + 5") 

= v(Tr zT,r;/2)(^(,),(,+i)(5)/)(r„r_,, + 5) 

(6.30) = M9KS), 
for all G 3r+i) and 

(w(r),(r+i)(exp^)/)r(c/)(5) = (a;(r),(r+i)(exp^)[tj(r)_(r+i)(5)/])(rr,T_r + 5) 

= {^ir),ir){-RTr)[(^(r),{r+l){9)f]{Tr)){T^r + S) 

= ^(Tr((T_, + 5)*i?r,))[a;(,),(,+i)(^^)/](r„r_, + 5) 

ig)f]{Tr,T^r + S) 

(6.31) = X7(expi?)/r(5)(5), 

for all G Hom(V^+i, V^r-i) ® ^-r) C Ur+i, in other words, for all u G f/r+i) 

i^ir),{r+l){u)f)r{9){S) = X^{u) fr{9){S) ■ 

It is then clear that understanding the map / i-^ is an important first step towards understanding 
the more complicated map / i-^ /(^j. 

We start by observing that G^j._i^Nr is the stabilizer of Tr in G. Hence, according to equation 
(j6.27p we obtain a representation (cTr) ^r),(r)) of G(^j._^i^Nr, where we are viewing as the 

fiber over the point Tr G Hom(V(^), V^+i). Now, since by definition (TTri9)f = ^{r),{r)i9)'-P foi' 
if G ^j.) (j.), we may use the formulas given in ^ to describe this action explicitly: given (p G (^), 
S G Hom(y_^,%)), 

(6.32) (CTT,(expZ)^)(5) = V(Tr 5*5Z/2)v9(5), for ah Z G 3^, 

(6.33) iaTAexpR)ip){S) = io^r-iUr)iSR)['piS)], for ah G Hom(y(,,_i), F_,), 

(6.34) {aT,{9)v){S) = L0(^r-i),ir){9)MS)], for ah g G 

Now, given ip G ^?^r),(r)) set ifi^S) = (p{T-r + S) for all 5 G Hom(y_r, V-r)- It is clear that the 
map ip ^ (p defines a surjection 

Furthermore, using equations (|6.32p - (j6.34p . we may thus define a representation ((Jt^, ^-r,-r ^ 
^r-i),(r)) of C!(^,f.~i)^r explicitly by the following formulas: given ip G ^-r.-r <8 ^r-i),(r)) G 
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Hom(y_r,K_^), 

(6.35) (aT,,(expZ)(^)(5) = V(Tr(T_, + 5)*(r_, + 5)Z/2)(/p(5) = (^(S), for Z G 3,, 

(6.36) (aT.(expi2)(/p)(5) = W(,_i),(,)((r_, + 5)i?)[^(5)], for G Hom(F(,_i), K.,), 

(6.37) (or, (£/)(/?) (5) = a;(r_i),(r)(c/)[y'(5')], for5GG(^_i). 



Observe that if / G ?^i^r),{r+i)j then fr{g) = [u}(^r),{r+i)i9) f]i'^r) ■ It follows immediately that 

/. G C°°(G(,„i)A^AG;^-r,-r ^ ^^r-l),(r))- 

Now, given p a seminorm in ^r-i) (r)) -^1 ^ ^(Hom(y_r, F_r)), .^2 ^ ^^(0); di, (i2 G N and 

/GC°°(G;^„,,_,^^^,_i),(,,)),let 

(6.38) qz„ZM2Af) = sup p(Zi[Z2/(mfc)](5))||mf 1(1 + WSW)"', 

k e K, m e Mr 
S G Hom(y_,., V-r) 

where ii' C G is a maximal compact subgroup as in Section 14.11 Set 

{/.C~(iV.G„_,AG;^-.-. I "^Jf L^^i^'" } ■ 

Lemma 6.5. The map f fr induces a G -intertwining isomorphism 

where NrG(^j._i^ acts on J/'-r -r ^ ^r-i) (r) representation aT^, given in equations \6. 35]) - 

mm- 

Proof. Let A G (^^^ (r+i))^'"''^''^''"''^- the Appendix, we wih identify A with the ^^^-^ ^.^^-valued 

distribution on Hom(V(^), V^+i) that sends / G =5^(r),r-+i to the linear functional v ^ A(/ ® v). In 
other words, and with some abuse of notation, we set 

\{f ){v) := X{f v) for all / G ^(r),r+i, v G %),(r)- 

Now recall that for Z G Ir+i, f £ =5^(r),r+i ^-^d T G Hom(V(r), ^+i), we have uj(r),{r+i)i^^P ^)f{T) = 
■tp{TT ZTT*/2)f{T). Hence, for all such Z we have that 

(6.39) ^{r),{r+i) (exp Z)X = V'(Tr ZTT*/2)X, 

that is, the action of exp Z on A is given by multiplication by the function T i-^ ^/^(Tr ZTT* /2). On 
the other hand, since A G (^^) (^r+i)^^''^^'^'^'^^ ^^'^ since 3^+1 C u^+i, we must have that 

(6.40) a;(r),(r+i)(expZ)A = ijj{TrZX/2)X = A, 

where the latter equality is due to the fact that Z G S-2r-2(^)i ^ G 02 (^) and r > 0. Define 
hz G C°°(Hom(F(^),y^+i)) by h^{T) = i;{Tr ZTT* /2) - 1 for all T G Hom(y(^), V;.+i). Then we 
can reformulate equations (I6.39P and (|6.40p by saying that h^X = 0, for all Z G 3r+i- It then follows 
immediately that 

suppA C Pi {T G Hom(V(^), V;+i) | /i^(T) = 0} 
= {T G Hom("\/(^), V;+i) I TT* = 0}. 

Let 

(6.41) TiomNMiV(r),yr+i) = {Te Hom(y(,), V;+i) | TT* = 0}, and 
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(6.42) HomG7VM(^(T-)) ^+i) = {T S Hom(V(j.), V^+i) | TT* = and T has maximal rank}. 

(Here the subscripts MN and GMN stand for nuU mappings and generic null mappings, respectively.) 
We claim that the natural map 

factors through the intermediate space =5^(HomcjArM(^{r)) K-+i)) ® ^r),{r)- other words, if A G 
(^^r) (r+i))^'^^^''^'^''^^' th^^ '^^'^ identify A with a ^^^^ j-^^-valued tempered distribution living on 
HomG'iVM(V(r)) ^+i)- So far we have only shown that if A € (^r) (r+i))^'^'''^''^'^'''^' then suppA C 
Horn ATM (V(r),V;+i)- Let 

^ = {T G Hom(V(j,), K-+i) I T is singular}. 

Clearly ilomGNM{V(r)T^r+i) = Hom7VM(V(r)5 K'+i) n^'^, where A'^ is the complement of ^. So, as 
a first step towards proving our claim, we will show that 

(6.43) if A G (^^) (^+;^))^'-+i'*'-+i and suppAc^, then A = 0. 

Observe that A C Hom(V(r), T^+i) is invariant under the natural action of G{V(^r)) ^ GL(T^+i) 
and decomposes as a finite union of orbits. (Actually, Hom(V(j.), V^+i) itself decomposes as a finite 
union of orbits.) Furthermore, if C C ^ is a closed invariant subset, then there exists an orbit O C C 
that is open in C. Therefore, we may use a Bruhat type argument to prove statement (|6.43p . To be 
precise, we will show that if A G (^^) (r+i))^'"^^'^'^'^^ ' C C ^ is a non-empty closed invariant subset 
containing supp A and O C C is a relatively open orbit, then A vanishes in the complement of C\0. 
Once this is done, a straightforward inductive argument finishes the proof of statement (j6.43p . 

If k is non-Archimedian then we may immediately identify A with a distribution living on C. 
In this case, by restricting A to O we obtain a well-defined distribution on this orbit. So we 
will just need to focus on the case where k is Archimedian. Given any x € O, there exists a 
neighborhood C Hom(V(j,), V^+i) of x such that the order of X\u^ is < / for some I G N. 
Therefore, we may find a locally finite open cover {J7i}igN of Hom(V(r), 14-+i) and a partition of 
unity {pi} subordinated to {Ui} such that A = piX and each piX is of order < /j, for some /j G N. 
Now, since multiplication by a C°°-function clearly commutes with the action of Ur+i, we have that, 
if A G (^^) (^+1))^''+' then piX is also in (^^^ (^+1))^"+''*''+' for all i. The upshot is that in 

order to prove statement (j6.43p it is enough to just consider distributions A G (^r) (r+i))'^'^^^''*^'^^^ 
of finite order. But in such a case, to prove the vanishing of A in the complement of C\0, it suffices 
to show that D'{0;iV(^r),{r) ® M(')')'^'-+1'*'-+i = for all / > [S]. Here M^') is the l-th transverse 
jet bundle of O (see the Appendix). Now observe that, given S G Hom(V(^), Vr+i) and Z G Ir+i, 



A(expZ)./(T) = I 



'0(TV(r + tS){T + tS)*Z/2)f{T + tS) 

t=o 

V'(TrTT*Z/2)V^(Tr5'r*Z/2)/(r) + V(TrTT*Z/2)^(Trr5'*Z/2)/(r) 
df 

+ ^{TtTT*Z/2)-^{T), 
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and if i? G Hom(l^-(_i, V(r)) C n^+i, then 



^(e.pii)/(T) = 



t=0 



d 

dt 



UJ(r),(r)iRT)uj(r),(r)iRtS)f{T + tS) 

i=0 

d 



What the above couple of equations imply is that the action of Nr-\-i (and in particular the action 

of Ur+i) on the l-th transverse bundle is trivial, and hence D'{0;i!V^r),{r) ^ {M^^^y)^-+^'^-+^ = 

M^') (g) Z)'(C'; ^^r) From all this we see that, in order to prove statement (j6.43p , it 

suffices to show that 

(6.44) D'{0; %-)Xr)f^+''^^+' = 
for every orbit O C A. 

At this point we come back to the general case, where k can be Archimedean or non- Archimedean. 
Let A G D'{0;^r),{r))^''^^'^''^\ where O C A is a GiV^r)) x GL(K.+i)-orbit. As we have already 
seen, the equivariance of A implies that suppA G O PI Hom7VA/(^(r)) Since the latter set 

is G(V(r)) X GL(Vr+i) invariant, we see that actually O C HomjvAf (V(,r), Ff+i). From equations 
and dSSS]), it is clear that once we fix any T G O we could describe C^{0) '3^(r),(r) 
as an induced representation. (Here we are using the identifications G = G(V(j,)) -^r+i — 
GL(V^_l_i).) Translating under the action of G(V(r)), if necessary, we may assume that T £ O 
satisfies T\^r-i = 0. To see why, let T G Homjvjvf (V(r.), V^r+i) {Q HomjvAf (Vj-^), V(f._|_i))), and so 
T*\y± = 0. Since = V(^r) © we have that ImT* (c V(^r)) is a totally isotropic subspace 

with dimension less than or equal to dimy_.f._i = dimV_r. We may thus assume, by translating 
under G{V(^r)) if necessary, that ImT* C V-r, which will imply that r|^r-i = 0. We identify 

such a T with an element of Hom(T4-, C Hom(V(j,-), K+i) in an obvious way. Now we have an 

isomorphism of G x Pf+i-modules: 

(6.45) C^{0) » %Ur) = - lnd'^^^^+'{%ur))- 

Here Stab^^ denotes the stabilizer of T in G x P^+i and the notation — Ind means that we are 
taking compactly supported induction. The isomorphism is given by the map / i— s- /, where we have 
set f{g,p) = [uj(^r),{r+i){9)^{r),(r+i){p)f]{T), for any given / G C^{0) (g) %),(r)- Its inverse is given 

by the map h^h, where h{p^^Tg) = aT{g)^^o-T{p)~'^h{g,p) for all h G — Indg^^^j^^^ (^^.r)^(.r)), 

g £ G, p £ Pr+i- Let us describe the action on the right hand side of equation (|6.45p more precisely. 
For this we will use the isomorphism ^^j.) = o5^_^ ^(^r-i),{r)- According to equations (j6.22p - 
(j6.28p . the action of G(j.)-^r+i on ^i^r),{r) is given by 

(6.46) aT{g)h{S) = ^(r-i),{r){g)h{a~^ S) forgGG'(.r), 

(6.47) c7T(expZ)/i(5) = ^^{Ty ZTT* /2)h{S) for Z G 

(6.48) cJT(expfi)/i(5) = u:i^r),{r){RT)h{S) = %b {Ti RTS*)h{S) for R G Hom(V;+i, %))• 

In the last equality, we have used the fact that, since T G Hom(V^, Vr+i) C Hom(V(r), Vr+i), then RT 

is in Hom(V^, V(j,)) C Hom(V(j.-), Vj-^)) which is a totally isotropic subspace that is complementary 
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to 'Rom{V-. r,V^r))- (Here is where the choice of T simphfies matters.) Now observe that there 
is a continuous G x Pr+i-intertwining map from C^{G x Pr+i x Hom(y_r, V(r)); ^^r-i),{r)) to 
— Indg^^^^ (o5^_j,_(j,-) (8) ^^r-i),(r))) with dense image, given by 

h{g,p,S)= / {aT{x)-^h{xgg,Xpp){S)dx, /i E x x Hom(y_,., %)); 

JStabr 

Here Xg and xp are the projections of x G Staby to G and Pr+i, respectively. Therefore, we may 
define a ?7j.+i-equivariant ^^-i) (r)"^^^^^*^ distribution A on G x P^+i x ilom{V-r,V(^r)) by setting 

A(/i) = A(^), for all h G C^{G x P^+i x Hom(y_^, 
It is clear that if p = mgh, with m G Mr+i = GL(14-+i), n G A^^+i, and ^ G then, for all 

heC^ - lnd^^^^+' {y^r,(r) ^ %-i),(r)), G Hom(y_^,%)), and R G Hom(V;+i, %)), we have 

exp R ■ h{g, rhgn, S) = ^(Tr Rm^^TS*g)h{g, rhgn, S). 

Hence 

exp ^ • A = ip{TT Rrh-^TS*g)X. 

In other words, the action of exp it! on A is given by multiplication by the function {g,mgh, S) i— )• 
ipiTr Rrh~^TS*g). On the other hand, since A is C/r+i-equivariant, we should have 

exp ^ • A = ipiTi RX)X, for all R G Hom(V;.+i, V(,,„i) V^,.) C u^+i- 

Here for the trace function, we are considering X and R as elements of End(V(j,^x))- Therefore, 

suppA C {{g,7hgh,S) \ Rrh^^TS*g = RX, for ah R G Rom{Vr+i,V(r-i) © V-r)}, 

but the latter set is empty since ImX contains V^+i, whereas the image of m^^TS*g does not have 
the same property due to the fact that T is singular. From all this, we conclude that if O C 
then L>'(C';^^)^(^))^'-+i'^'-+i = 0. This is statement (fOill which, as we have already seen, implies 
statement (j6.43p . 

Having proved statement (j6.43p . we come back to our original assertion that the map ^^j.) 
i'^{r),{r+i))ijr+i xr+i f^^tors through S^{Yiom.GNM{y{T)i^r+i)) ® ^r),{r)- We have seen that any 
A G (^(^) (r+i))^'^^^'*'^^^ is completely determined by its restriction to C'^{A^), where A*^ is the 
complement of A in Hom(V(j.), l^+i). But on A^, is a regular value of the map T ^ TT*. 
Therefore, X\a': is a distribution that lives on HomcjvAf (V(^), V^+i), that is, A can be identified with 
an element of (=5^(HomG'jVAf(^(r); ^+i)) © ^r),(r))') we wanted to show. 

Given / G ^{'HomGNM{V(^r)-,Vr+i)) (S> '3^(^r),{r), we set 

(6-49) fr{g) = [uj(^r),{r+i){9)f]{Tr) = uj(^r),ir){9)[f (Trg)], for g £G. 

Then it is clear that fr G C°°{NrG(^^_i)\G;&^(^,.),ir)) for ah / G y(HomGNM{Vir),Vr+i)) O 
Let 

^+(7V.G(,,_i)\G; %),(,,)) = {/. I / G yi}iomGNMiV(^r),Vr+l)) ^(rUr)}- 

Now observe that HomGArA,/(^(r)) ^+i) is a single orbit under the action of G. Therefore, since 
G(r-i)-^r is the stabilizer of G HomGjv7v/(^(r)) we have that as G-modules 

^{RomGNM{V(^r),Vr+i)) %,),(,,) ^ ^+(iV,G(,„i)\G; %),(.))• 
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Now we will show that 

(6.50) ^'-{NrG^,_,^\G;^^^,)^^,))^^^^^^^^^^y+{NrG 
where 

So let A G (=^+(A^rG(,._i)\G;^^)^(^))')^'-+^'^'-+^ We will identify A with a (^-j-valued dis- 

tribution on G X Hom(y_r, V(r)). Observe that, for all / G =5^+(A''rG(^_i)\G; ^^^) (^)), g & G, 
S G }iom{V-r,y{r))j E^iid R S Hom(\4-+i, V(r))j we have 

expi? • = V'(Tr i?r,,5*)/(5)(5). 

Therefore, since A G (=y+(7VrG(r_i)\G; ^(^)^(^))')^■'+^'^■'+^ we must have 

i^{TrRTrS*)X = iP{TtRX)X, 

for all R G Hom(T44_i, V(^_i) V-r) C Ur+i- Now, since X is a regular value of the map S i— )• Tj-S*, 
we conclude that A is a distribution that lives on 

Gx{S e Hom(y_^, %)) I RTrS* = RX for all R G Hom(V;+i, T^-r)}- 

But the last set is just G x [T_r + Hom(VLr5 ^-r)]- The isomorphism given in equation (|6.50p now 
follows immediately. 

The only thing that remains to be checked is that 

(6.51) y+{NrG(^r-l)\G; y-r,-r O %-l),{r)) = '^(A^rG'(r~l)\G; ^-..-r ® %-l),(r))- 

For this, let p he a, seminorm in ^(^r-i),{r)i ^ ^(Hom(y_^., V.^)); Z2 G U{q), di, ^2 G N and 

/ G y(r),r+l ^ y~r,{r) ^ %-l),{r)- Then 

'7Zi,Z2,di,d2,p(/r) = sup p{Zi[RzJr{mk)]{Smmk\f^l + \\S\\)''' 

k£K,meMr,SeUom{V-r,V-r) 



sup /5(Zi[(A: ■ RzJ)r{m)]{SmmrHl + ||5||) 



c^i n ^ II Qlh'^2 

(^in _L II Qlh'^2 



= sup /5(a;(,),(,)(m)Zi(A:-i?zJ)(T.m)(r_, + 5))||mri(l + ll'5|l) 

A;,r?i,5 

= sup /5(Zi(A;-i?z2/)(T,m)(r_,m + 5m))||mfHl+ ll^ll)'^^ 

k,m,S 

< sup qzi,ci,c2,p{k ■ -Rz2/)(1 + ll^'r'n- + r.^mlD'^^ 

k,m,S 

■ {l + \\Sm\\)-'''\\mf'{l + \\S\\f\ 

where ci ,C2 G N are arbitrary and qzi,ci,c2,p is a seminorm on ,,+1 (8) ^^r-i),(r)- But 

then there exists a constant C > such that 



qz^ZM^Afr) < sup Cgzi,ci,c2,p(A:-i?zJ)||m|I-^Hl + ll^ll)"''ll"^ir'll"^fHl +11^11) 



-C2||^||C2||^||din ^ II Q|hrf2 
dl+C2-Cl/-| I ||Q||^,ci2-C2 



< sup Cqz,,c,,c2Ak-RzJ)\\mr^^''-''{l + \\S\\) 

k,m,,S 

Now observe that if we fix C2 ^ (i2 and ci ^ di + C2, then 

supCgz„ei,c2,p(A: • RzJ)\\m\f^+'''^-^^ (1 + H^ID'^^-^^ < 00, 

m,S 
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for all k £ K. Furthermore, the map k ^ sup^ 5 CgfZi,ci,c2,p(^ • (1 + ||5'||)'^2-C2 jg 

continuous, and hence, since K is compact, bounded. Therefore qzi,Z2,di,d2,pifr) < 00 and (I6.5ip 
follows. □ 

Observe that the space (^?^r),(r+i))[7 ^ carries a natural action of the group N. As a con- 
sequence of the isomorphism (^r),(r+i))c/ +1 x +1 ~ '^i^rG(^r-i)\G', ^S^-r-r ^r-i),(r))) may 
define an action of on '^{NrG(^r-i)\G; ^~r,~r ^(r-i),(r)) by the formula (n • fr){g){S) := 
(w(r),(r+i)(^)/)-r(<7)(5'), for / € ^(^r),(r+i)i n £ N and S G ILom{V-r , V-r) ■ We describe this action 
more concretely in the following 

Lemma 6.6. There is an induced action of N (= N(,^^Nr+i) on'^{NrGi^r_i-^\G; ,y-r.-r^'S^(r-i),(r))j 
as follows: for all (p £ ^{NrG(^r-i)\G; y-r-r ® '3^{r-i),{r))' g &G, S £ Rom{V-r,V-r), 

{expZ-(t>){g){S) = ^{g){S) forZ£jr+i 

{e^pR-(t>){g){S) = iP{Tv RTrT%)tlj{TT RTrS*)(l){g){S) /or i? G Hom(t>+i, %)) 

{exp Z ■ (t)){g){S) = a;(,._i)_(r)(expZ)[0(5r)(5')] for Z £ jr 

{expR-ct>){g){S) = a;(,_i),(,)(exp i?)[</.(5)(5 + i?*r_,)] /or G Hom(V;, 

{n ■ (t>){9){S) = u}(^r-i),{r){rt)[4>{9){S)] for h £ . 
In particular, we have 

(6.52) iii-<P)ig)iS)=u;^r-iUr)mHg)iS)] foru£U^,y 

Proof li Z £ Ir+i, we have already seen in equation (I6.30p that 

(W(r),(r+l)(expZ)/),.(5)(5) = fr{g){S), 
while if G Hom(I4-+i, V(^r)) C ftr+ii similar to equation (j6.3ip . we have 

(w(,),(,+i)(expi?)/),(5)(5) = i;{TrRTriT.r + Sr)fr{g){S) 

= ^l,{T,RTrT%)ilj{TTRTrS*)fr{g){S). 

On the other hand, if Z £ jr, then 

(w(r),(r+i)(expZ)/)r(g)(5) = (L^(^)_(^+i)(expZ)[a;(^)^(^+i)(5r)/])(rr,T_r + 5) 

= (^(r),(r)(exp^)[(a;(r),(r+i)(5)/)(7;)])(T_^ + S) 
= W(r_i),(r)(expZ)([a;(r)_(^+i)(5r)/](T,,(expZ)~^(r_r + S))) 
= u;(r_i),(^)(expZ)([a;(,.)_(,.+i)(5r)/](T,,r_, + S)) 
= a;(r_i),(r)(expZ)/r(5)(5'). 

Here we are using that Z(T^r + S) = because Im(T_,. + 5) C V-r+i © ^-r and Z\y ^^^y = 0. 
On the other hand, if G Hom(V^, V(j._i)) C fir, then, according to equations (|6.46p . (j6.47p and 

Hr),(r+i)(exp^)/)r(5)(5') = (w(r),(r+i) (exp ^) (fii)/] ) (T^ , + S") 

= {^{r),{r){eXVR)[{^^{r),{r+l){9)f){Tr)]){T-r + S) 
^(r— l),(r) 

(expi?)([a;(,),(,+i)(5)/](T,, (exp i?)-i(r_, + 5))) 

= UJ(_r_i)^(^r){ewR){[^{r),{r+l)i9)f]{Tr,T^r + S + R*T-r)) 

= ^(r-l),(r)(exp-R)/r(c/)('S' + R*T^r)- 
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Here we are using the identification Hom(V^, V(^r-i)) B R R — R* & fir bs in equation (I6.20p . We 
are also using that, since T_,. E Hom(y_r, l^_r+i), then R*T^r G Hom(K_.f., and that 

R* S^r — RS—r = RT—r — R* RT—r — R*RS—j- = 0, 
for all R € Hom(T4, V(^_i)), 5 € Hom(y_.r, ^-r)- We note, in particular, that R*T-.r = if 
R G Rom{Vr, V{r-2) ® ^-r+i) C Ur- Finally, if n G ^(r-i) C then 

{^{r),{r+l){n)f)r{g){S) = (W(r),(r+1) (^) Kr),(r+1) (5)/] ) , + S) 

= UJ(_r-l),{r){^){['^{r),(r+l){g)f]{Tr,n^^{T-r + S))) 
= ^{r~l),{r){n){[i^[r),{r+l)i9)f]{Tr,T^r + S)) 
= i^{r-l)Xr){fl)fr{g){S). 

Here we are using that, considered as an element of G(^), n G Nf^^-i) leaves V_r-+i ® ^-r fixed. □ 

6.5. From 'S/'^j.-^^^^^+i) to =^,7: proof of Proposition [6T4l We are finally ready to complete the 
proof of the key proposition, to which we refer the reader for the unexplained notation below. 

Proof (of Proposition \ 6.4^ . We will prove the result by induction on r. The case r = is immediate. 
Observe that U = Ui^j.)Ur+i a^nd X(r+i) = X(r)Xr+i- Therefore, from Lemma 16.51 

where the last equation follows from the fact that U(^r) acts only on the values. See equation (j6.52p . 
Now, given / G S^-r-r ^(r-i),(r)) 9 ^ G and S^k £ Hom(y_fc, VLfc), A; = 1, set 

f(r-l)ig)iS-r, S-l) = {aTr{9)f){S-r,Tr-l,T-r+l + S-r+1, •••,?()) 

= {LO(^r-l),{r)ig)[f{S-r)]){Tr-l,T^r+l + 5'-r+l, • • • ,2o). 

For convenience, we denote the representation r^- on (defined in equation (|6.7p ) by the 

more concise symbol J^t- Then, by induction hypothesis, the map / 1— t- /(r-i) induces a 
intertwining isomorphism 

where T^i^ = ®\^_iT}^, for Z = 0, . . . , r. Notice that T^^,) = T^,^. Also notice that in the above equa- 
tion we are identifying the spaces <^ '^(-^(r-i)\G'(r-i)j — '^{^[r-i)\G{r-i)] '^^^r))- 
We claim that ^(r-i) is actually an isomorphism of A^^-modules: 

(6.53) :^_,,_,0(^,_i),(,))^^^^^-^^^ ^^(iV\G(,_i)iV,;jrT(^,), 
where 

^(iV\G(,_i)iV,;J^T„) = {/ G C-(iV\G(,_i)iV,; JTt^^j) |/|G(„_„ G ^(iV(.-i)\G(,_i); ^t,..,,).} 

Observe that if / G ^(iV\G(r_i) A'^r; ^(^) ), 5 G ^(r^i) and n G Ar, then 

(ra • f){9) = f{9n) = f{{9ng~^)g) = TT^^^{gng'^)f{g). 

Therefore, to prove our claim, we need to show that for all / G S^-r.-r ® ^r-i),(r)) 9 ^ G'(r-i)> 
n G A'r, 

(6.54) (aT,(f^)/)(r-i)(5) = -rT(,)(5?^5~^)/(r-i)(5)- 
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Now, by definition, 

{^Tr(.n)f)(^r-l){9){S-r, ■ ■ -,3^1) = {aTr{g)o-Tr{n)f){S-r,Tr-l,T_r+l + S-r+1, • • • ,7o) 

= {^Tr{9ng~^)[aTr{9)f]){S-r,Tr-l,T^r+l + S-r+1,--- ,To) 

= {^Tr{9ng~^)[^Tr{9)f]){r-i){e){S-r, S-r+l, • • • , S-i). 
From all this we see that in order to prove the claim given in equation (j6.53p it suffices to show that 

for all / € y^r-r ^r-l),(r)i ^ ^r, 

(6-55) (o^r,(n)/)(r-i)(e) = rr,^)(?^)/(r-i)(e)- 

Now, by equation (j6.35p . if Z € 3^, then 

(aT,(expZ)/)(,._i)(e)(S'_r., . . .,S^i) = f{S^r,Tr-i, . . . ,To) = /(^_i)(e)(S'_,., . . .,S-i). 
On the other hand, by equation (j6.36p . if i? G Hom(V(^„2) ® Vr-i, V-r) C Ur, then 

{aTr{ewR)f){r-i){e){S-r, ■ ■ • ,5'_i) = (a;(r_i)^(^)(T_rii)[/(S'_,.)])(^r-i, • • • ,7o) 

= v(- Tr r_,i?r;_2)/(5_,, . . . , Tq) 

= X7(exp-R)"V(r-i)(e)(5'_r., . . . , 

in other words, for all u Ur, 

{^Tr{u)f)(r-l){e){S-r, . . . , S-i) = X-y{uy^f{r-l){e){S-.r, S-i). 

Finally, again by equation (|6.36p . if i? € 0-i fl — Hom(y_r+i) V-r), then 

{aT,{e^VR)f){r-l){e){S-r, ■ ■ • ,5'_i) = {uj(^r-l),{r){T-rR + S ^rR)[f {S -r)]{Tr-l, ■ ■ ■ ,Tq) 

= Tr 5_,i2r;_i)/(,_i)(e)(5_,, + r_,i?, . . . , 

Now, for such an R, ax^^^^iR, 0) = T^rR — Ty—iR* . Therefore, 

TT^^r) (expi?)/(r-l)(e)(5_r, • • • , ^-i) = Ty^y{T_rR - Tr-lR*)f(^r-l){e){S^r, • • • , -S*-!) 

= 'ip{-Ti:S^rRT*^l)f{r-l)ie)iS-r,S-r+l + T^rR, ■ ■ -,3-1). 

Therefore, equation (j6.55p holds true, and hence we have proved our claim given in equation (j6.53p . 
We will now show that 

Given / G '^{NrG^r-i)\G;'rf{N\G^r-i)Nr;J^T^^^)), set f{g) = f{g){e), for ah g e G. Then it is 
clear that / G C°^{N\G; J^t^,^), but we claim that / is actually in 'if{N\G;J^T^,^). Effectively, 
given a semi-norm p of J^^^y Z G U{q) and d G N, 

qzApif) = sup p{Rzf{m2mki))\\mim2\\'^ 

kaK, mi^Mr, m2£G(^_i) 

= sup p{Rzf{'m'2mki){e))\\mim2\f 

k,m\ ,m2 

< sup p{Rzf{mki){m2'^))\\mi\\'^\\m2\\'^ 

k,m\ ,m2 
= PZA,d,d,pif) < 

Now given / G ^(A^\G; J^^^.^ ), set f{g){h) = f{hg). Then again it is clear that 

/ G C°°(iV,G(,_i)\G;C"»(iV\iV,G(,_i); JTt^^,)), 

but we claim that, actually, / is a function in ^{NrG{y.-i)\G]'^{N\Gi^r^i'^Nr]<^T(^)))- Effectively, 
let /3 be a semi-norm of Zi G U{q), Z2 G f^(0(r-i)) and di, ^2 G N. Then Z2 G U"^{q) for some 
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m and hence there exists a basis {Yj^j^^ of U^{q) such that for ah g ^ G, Ad{g)Z2 = aj{g)Yj for 
some functions aj. Since (Ad, ?7'"(0)) is a finite dimensional representation, there exists a constant 
dm such that \a{g)\ < WgW^"" for all g ^ G. Taking this into account we have that 



PZuZ2,dx,d2,pU) = p{Rz2{Rz^f{mki)){m2))\\mi\\ ^\\m2 

k&K, migAfr, m2€:G'(^_i) 

= sup p{RAd{mki)-^Z2RzJ{m2mki))\\mi\\'^^\\m2\\'^^ 

k,mi ,m2 

I 

< sup '^p{a{mki)RYjRzif{mkim2))\\mi\f^\\m2\\'''^ 



\d2 



k,mi,m2 

I 



d2 



< sup ^/0(fiy^.i?Zi/("iA;im2))||mifi+'^'"||m2| 

fc, mi ,7712 

I 

< sup ^p(i?K,.Zi/(mA;im2))||mim2f^+^™+'^2 

k,mi,m2 
I 

= '^QYjZi,di+d2+d^Af) < 

i=i 

Here we have used the fact that ||mim2|| = max{||mi||, ||m2||} > 11^2 1|- Now the only thing 

left to be check is that / = / and / = /, but by definition 

horn = figh) = f{gh){e) = f{g){h), 

and similarly f = f- □ 

Appendix A. Some facts on vector valued distributions 

In this appendix, we collect some facts about vector valued distributions. For a more detailed 
presentation of the material, the interested reader should refer to | il4j . over which this appendix is 
based. 



A.l. Vector valued distributions. Let A:" be a C°°-manifold and ^ a Frechet space. We write 
G°°{X; and G^{X; 'f) for the topological vector spaces of the C°° mappings from X to and 
those with compact support, respectively. Let i^' be the dual of Y provided with the strong dual 
topology, that is, the topology of uniform convergence on bounded subsets of Y. Finally, let Diff (Af) 
be the sheaf of differential operators on X. 

Definition. A ^^-distribution on X is an element of the topological vector space 

D'{X; r) := G^{X- Y)' ^ {G^{X) ^ T)' 

consisting of the continuous linear forms on G^{X; Y), provided with the strong dual topology. A 
"^'-valued distribution on X is an element of the space Lin(C^(A'); Y') of continuous linear maps 
from C^iX) to Y' . We wih also equip Lin(C^°°(A'); T') with the strong dual topology. 

A Y distribution A defines canonically a "^'-valued distribution A on X, that is, we have a 
canonical isomorphism 

A^A D'{X-y) ^Uii{G^{X);y'). 
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In fact, if A e D'{X; t) and / G C^[X), then A(/) G is well-defined by A(/)(v) = A(/ ® v), for 
sl\ V . In what follows we will frequently identify these two spaces. 



A.2. Transverse order of a ^-distribution. 

Definition. A '^-distribution A € D'iX^Y) is said to be of order < I if for every compact set 
K, d X there exists a constant c > 0, Zi, . . . , £ Diff^^^(^) and a seminorm v onY such that for 
all/GCr(A';r) 

m 

|A(/)|<c^supK^,/(a;)). 

We write D'^^\X; Y) for the subspace of D'{X-, y) consisting of elements of order < I. 

Note that every "^-distribution on X is of finite order over every relatively compact open subset 
oiX. 

Suppose C is a closed C°°-manifold that is regularly embedded in X. We write D'^{X; for the 
distributions in D'(X;i^) with support contained in C. We say that A € D'{X;y) has transverse 
order < Z at a; G C if the following holds: there exists an open neighborhood Ux of a; in Af such that 

iifeC^{Ux;y) satisfies Zf\cnu. = 
^^•f)-^^ for all ZGDiffW ([/,). 

If A satisfies this definition, then the transverse order of A is < Z at all points of C in a suitable 
neighborhood of x. Let D'^''\X;Y) be the linear subspace of elements in D'{X;'y) which have 
transverse order < I for all points of C. We say that A lives on C if A belongs to D'(C; Y) := 

DfXx-r). This is the same as saying that A(/) = if / G C^iX^y) satisfies f\c = 0. In this 
case there exists a unique distribution ji on C such that A(/) = fi{f\c), for ah / G C^{X; Y); and 

often we will identify A with ^u. Note that in this case A G D'^^\X\ Y) and supp(A) C C. 



A.3. Transverse jet bundle. As before, let A' be a C°°-manifold, and let C C be a closed C°°- 
manifold regularly embedded in X . For any x G C. let be the algebra of germs of C°° functions 

defined around x and Diff^''' the C^-module of germs of differential operators of order < I defined 
around x. Let Vx'' be the C^-submodule of Diff^^'' generated by germs of ^-tuples (ui, . . . ,vi) of 
vector fields around x for which at least one of the ViS is tangent to C; let 4'^ = Diffi' +V^^\ 
By choosing local coordinates at x, it is easy to see that Ix^ actually is the stalk at a; of a subsheaf 
j(') of the sheaf DifF^'^ of differential operators of order < I on X. Hence we have a well-defined 
quotient sheaf 

M«=DiffW //«, 

with stalk at x equal to Mx = Diffi; / Ix ■ We write a i-^' a for the projection Diffi: ' — > Mx ■ 

(r) (r) (r) 

Since Diffx and Ix are both C^-modules, Mx also has the structure of a C^-module, and the 
projection is a mapping of C^-modules. 

Given x G C we can select an open neighborhood Ux of X and local coordinates {t : u) = 
{ti, . . . ,tp,ui, . . . , Uq) on Ux such that 

UxnC = {{t:u)\ti = ... = tp = 0}. 

It is easy to show that the elements 3" (|a| = I) form a free basis for the sections of M^') around 
X. Here a = {ai, . . . , ap) is a multi-index, |a| = ai H \- ap and 5" = d^"'^/d^^ ■ ■ ■ d^''. Thus M^'^ 
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is a vector bundle over C of finite rank. This is the l-th graded part of the transverse jet bundle on 
C. We have that M^'^ is the l-th symmetric power of M(i). We write M^)' for the dual to M^). 

A.4. Transverse symbols. In view of the isomorphism M^') {C^{C) Y)' = {C^{C) {M^'-y (g) 
(g)^))' it follows that 

D'{C; r) ^ D'{C] M^^y f ). 

Let T G Dp{X; T). Given x € C choose a neighborhood [/^^ of a; and a system of local coordinates 
(t : n) as before. Then there exist uniquely determined distributions G D'(C; Y) with |a| < 
such that on Ux we have 

(A.l) T= ^(-l)Hr„ar. 

|a|</ 

Theorem. For any T € Df\X] Y) there is a a{T) = a^^\T) G D'{C; M^')' 0^) with the following 
property: ifT has an expression in local coordinates as in equation iA.l\) . then 

a{T)= Y^df^To^. 

\a\=l 

The element cr(T) is uniquely determined by this requirement. The map 

a : Df{X; Y) D'{C; M«' Y) 

is injective modulo the linear subspace D'^^ ^\X;Y) and is independent of the choice of local coor- 
dinate systems. 
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